Skip to content

Commit 542e55a

Browse files
author
Chris Li
committed
initial repository
0 parents  commit 542e55a

File tree

220 files changed

+24735
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

220 files changed

+24735
-0
lines changed

‎.codecov_bash

Lines changed: 1285 additions & 0 deletions
Large diffs are not rendered by default.

‎.github/PULL_REQUEST_TEMPLATE.md

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
Dear Gobblin maintainers,
2+
3+
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
4+
5+
6+
### JIRA
7+
- [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
8+
- https://issues.apache.org/jira/browse/GOBBLIN-XXX
9+
10+
11+
### Description
12+
- [ ] Here are some details about my PR, including screenshots (if applicable):
13+
14+
15+
### Tests
16+
- [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
17+
18+
19+
### Commits
20+
- [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
21+
1. Subject is separated from body by a blank line
22+
2. Subject is limited to 50 characters
23+
3. Subject does not end with a period
24+
4. Subject uses the imperative mood ("add", not "adding")
25+
5. Body wraps at 72 characters
26+
6. Body explains "what" and "why", not "how"
27+

‎.github/workflows/build_and_test.yaml

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
# Copyright 2021 LinkedIn Corporation. All rights reserved.
2+
# Licensed under the BSD-2 Clause license.
3+
# See LICENSE in the project root for license information.
4+
5+
name: Build and Run Tests
6+
7+
on:
8+
push:
9+
# Publish only on `master`
10+
branches:
11+
- master
12+
pull_request:
13+
branches:
14+
- master
15+
release:
16+
types: [published, edited]
17+
18+
jobs:
19+
build:
20+
name: Build repository
21+
runs-on: ubuntu-latest
22+
steps:
23+
- name: Check out the repo
24+
uses: actions/checkout@v2
25+
- name: Set up JDK 1.8
26+
uses: actions/setup-java@v1
27+
with:
28+
java-version: 1.8
29+
# Stores external dependencies, can be further improved with Gradle 6.1
30+
- name: Cache Gradle Dependencies
31+
uses: actions/cache@v2
32+
with:
33+
path: |
34+
~/.gradle/caches
35+
~/.gradle/wrapper
36+
# Only rebuild cache if build.gradle is changed
37+
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }}
38+
restore-keys: ${{ runner.os }}-gradle
39+
- name: Build repository
40+
run: |
41+
./gradlew --no-daemon clean build -x test -x javadoc -x findbugsMain -x findbugsTest -x checkstyleMain -x checkstyleJmh -x checkstyleTest -Dorg.gradle.parallel=true
42+
43+
test_coverage:
44+
runs-on: ubuntu-latest
45+
name: Generate test coverage
46+
needs: build
47+
steps:
48+
- name: Check out the repo
49+
uses: actions/checkout@v2
50+
- name: Set up JDK 1.8
51+
uses: actions/setup-java@v1
52+
with:
53+
java-version: 1.8
54+
- name: Cache Gradle Dependencies
55+
uses: actions/cache@v2
56+
with:
57+
path: |
58+
~/.gradle/caches
59+
~/.gradle/wrapper
60+
# Only rebuild cache if build.gradle is changed
61+
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }}
62+
restore-keys: ${{ runner.os }}-gradle
63+
- name: Generate code coverage
64+
run: |
65+
./gradlew -PskipTestGroup=disabledOnCI -Dorg.gradle.parallel=false -DjacocoBuild=1 $GOBBLIN_GRADLE_OPTS jacocoTestCoverage
66+
67+
static_checks:
68+
name: Run static checks
69+
runs-on: ubuntu-latest
70+
needs: build
71+
steps:
72+
- name: Check out the repo
73+
uses: actions/checkout@v2
74+
- name: Set up JDK 1.8
75+
uses: actions/setup-java@v1
76+
with:
77+
java-version: 1.8
78+
# Stores external dependencies, can be further improved with Gradle 6.1
79+
- name: Cache Gradle Dependencies
80+
uses: actions/cache@v2
81+
with:
82+
path: |
83+
~/.gradle/caches
84+
~/.gradle/wrapper
85+
# Only rebuild cache if build.gradle is changed
86+
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }}
87+
restore-keys: ${{ runner.os }}-gradle
88+
- name: Run CheckStyle and FindBugs
89+
run: |
90+
./gradlew --no-daemon -x javadoc findbugsMain checkstyleMain checkstyleTest checkstyleJmh
91+
92+
run_tests:
93+
timeout-minutes: 60
94+
env:
95+
GOBBLIN_GRADLE_OPTS: "--no-daemon -Dgobblin.metastore.testing.embeddedMysqlEnabled=false -PusePreinstalledMysql=true"
96+
strategy:
97+
matrix:
98+
test-group: ["Core Tests", "Service Tests", "Module Tests", "Other Tests"]
99+
fail-fast: false
100+
runs-on: ubuntu-latest
101+
needs: build
102+
services:
103+
mysql:
104+
image: mysql:5.7.32
105+
env:
106+
MYSQL_USER: testUser
107+
MYSQL_PASSWORD: testPassword
108+
MYSQL_DATABASE: test
109+
MYSQL_ROOT_PASSWORD: password
110+
ports:
111+
- 3306:3306
112+
options: --health-cmd="mysqladmin ping" --health-interval=10s --health-timeout=5s --health-retries=5
113+
steps:
114+
- name: Check out the repo
115+
uses: actions/checkout@v2
116+
- name: Set up JDK 1.8
117+
uses: actions/setup-java@v1
118+
with:
119+
java-version: 1.8
120+
- name: Verify mysql connection
121+
run: |
122+
sudo apt-get install -y mysql-client
123+
mysql --host 127.0.0.1 --port 3306 -uroot -ppassword -e "SHOW DATABASES"
124+
- name: Cache Gradle Dependencies
125+
uses: actions/cache@v2
126+
with:
127+
path: |
128+
~/.gradle/caches
129+
~/.gradle/wrapper
130+
# Only rebuild cache if build.gradle is changed
131+
key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }}
132+
restore-keys: ${{ runner.os }}-gradle
133+
- name: Run test group ${{ matrix.test-group }}
134+
# Write retry logic as integration tests can fail due to timing out or network problems
135+
run: |
136+
./gradlew getGroupedTests -PgroupName="${{matrix.test-group}}" > temp.txt
137+
TASKS=$(sed -n 's/CI Task: //p' temp.txt)
138+
echo $TASKS
139+
140+
n=0
141+
until [ "$n" -ge 3 ]
142+
do
143+
./gradlew -PskipTestGroup=disabledOnCI $GOBBLIN_GRADLE_OPTS $TASKS -Dorg.gradle.parallel=false && break
144+
n=$((n+1))
145+
if [[ $n -lt 3 ]]; then
146+
echo "Tests failed, retry attempt number $n"
147+
else
148+
exit 1
149+
fi
150+
sleep 10
151+
done

‎.github/workflows/docker_build_publish.yaml

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# Copyright 2021 LinkedIn Corporation. All rights reserved.
2+
# Licensed under the BSD-2 Clause license.
3+
# See LICENSE in the project root for license information.
4+
5+
name: Build and Publish Docker image
6+
on:
7+
push:
8+
# Publish only on `master`
9+
branches:
10+
- master
11+
pull_request:
12+
branches:
13+
- master
14+
paths:
15+
- 'gobblin-docker/**'
16+
- '.github/workflows/docker_build_publish.yaml'
17+
release:
18+
types: [published, edited]
19+
20+
env:
21+
IMAGE_NAME: apache/gobblin
22+
23+
jobs:
24+
build_and_push_to_registry:
25+
name: Build docker images and publish to DockerHub
26+
runs-on: ubuntu-latest
27+
steps:
28+
- name: Check out the repo
29+
uses: actions/checkout@v2
30+
- name: Build Docker Tag
31+
id: build_tag
32+
run: |
33+
SHA=`echo ${{ github.sha }} | head -c 7`
34+
if [[ ${{ github.event_name }} == 'release' ]]; then
35+
TAG="${{ env.IMAGE_NAME }}:sha-$SHA, ${{ env.IMAGE_NAME }}:${{ github.event.release.tag_name }}, ${{ env.IMAGE_NAME }}:latest"
36+
else
37+
TAG="${{ env.IMAGE_NAME }}:sha-$SHA"
38+
fi
39+
echo "tag=$TAG"
40+
echo "::set-output name=tag::$TAG"
41+
- name: Set up Docker Buildx
42+
uses: docker/setup-buildx-action@v1
43+
- name: Login to DockerHub
44+
uses: docker/login-action@v1
45+
if: github.event_name != 'pull_request'
46+
with:
47+
username: ${{ secrets.DOCKERHUB_USER }}
48+
password: ${{ secrets.DOCKERHUB_TOKEN }}
49+
# - name: Login to GitHub Container Registry
50+
# if: github.event_name != 'pull_request'
51+
# uses: docker/login-action@v1
52+
# with:
53+
# registry: ghcr.io
54+
# username: ${{ github.repository_owner }}
55+
# password: ${{ secrets.CR_PAT }}
56+
- name: Build Images and Publish
57+
uses: docker/build-push-action@v2
58+
with:
59+
tags: ${{ steps.build_tag.outputs.tag }}
60+
push: ${{ github.event_name == 'release' }}
61+
file: ./gobblin-docker/gobblin/alpine-gobblin-latest/Dockerfile

‎.gitignore

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
.classpath*
2+
.project*
3+
.settings
4+
.DS_Store
5+
*.tar.gz
6+
7+
# Intellij related
8+
**/.idea
9+
**/*.iml
10+
**/*.iws
11+
**/*.ipr
12+
.shelf/
13+
.ideaDataSources/
14+
15+
# VS Code related
16+
.vscode
17+
18+
**/.classpath
19+
**/.project
20+
**/.settings
21+
22+
**/*.swp
23+
**/*.swo
24+
**/*.log
25+
26+
**/build/
27+
.gradle
28+
**/.gradle
29+
gradle.properties.release
30+
test-output
31+
**/test-output
32+
dist
33+
target
34+
tmp
35+
out
36+
**/out
37+
output
38+
/eclipse_build
39+
.project
40+
.classpath
41+
out/
42+
*/bin/
43+
**/mainGeneratedDataTemplate
44+
**/mainGeneratedRest
45+
**/main/snapshot
46+
**.tar.gz
47+
*~
48+
metastore_db/
49+
50+
# generated nodeJs files
51+
node_modules/
52+
package-lock.json
53+
54+
*.out
55+
56+
# generated java files
57+
**/gen-java/
58+
59+
temp/

‎.travis.yml

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
language: java
2+
3+
dist: bionic
4+
sudo: required
5+
6+
jdk:
7+
- openjdk8
8+
9+
addons:
10+
apt:
11+
packages:
12+
- libaio-dev
13+
- libdbus-glib-1-dev
14+
- xsltproc
15+
16+
before_cache:
17+
- rm -f $HOME/.gradle/caches/modules-2/modules-2.lock
18+
19+
cache:
20+
directories:
21+
- $HOME/.gradle/caches/
22+
- $HOME/.gradle/wrapper/
23+
24+
before_install:
25+
26+
services:
27+
- xvfb
28+
- mysql
29+
30+
stages:
31+
- test
32+
- name: deploy
33+
if: branch = master
34+
35+
before_script:
36+
- mysql -uroot -e "create user testUser identified by 'testPassword';"
37+
- mysql -uroot -e "SET PASSWORD FOR 'root'@'localhost' = PASSWORD('password')"
38+
39+
script:
40+
- travis_retry ./travis/test.sh
41+
- travis_retry ./gradlew jacocoTestReport
42+
after_success:
43+
- bash <(cat .codecov_bash)
44+
after_failure: ./travis/junit-errors-to-stdout.sh
45+
46+
jobs:
47+
include:
48+
- stage: deploy
49+
env: RUN_TEST_GROUP=none
50+
install: skip
51+
script:
52+
- travis_retry ./travis/bintrayDeploy.sh
53+
54+
env:
55+
jobs:
56+
- RUN_TEST_GROUP=build
57+
- RUN_TEST_GROUP=default
58+
- RUN_TEST_GROUP=group1
59+
- RUN_TEST_GROUP=coverage
60+
global:
61+
- secure: U72nmzXq7kXcIabiwvrTF+WkNFQxov2ACd8oPxWSHsyzRsfVJN42vT7gS3dLbH5G5claXG2p+rur4ueVffzYSwJ8B9OP6gTB8sNJnGr9zyxni4OJchyKqOYuj+UBpEQC/7qXKMCKnuJndsf1OvndDh/V1SH0DSSUuA6mDtgO/eM=
62+
- secure: WiK7tyFV68xdkIfUlWreUHgEGGjaCBd73O4SbjE9AsbqqF7D+Iu8iRo1OhKQj+6eajUH9Eoev9rVN74FQgUfeNzrOkYsgDysXmyZ7+UxFokijFcATJmIBompA3dySGU2qXeKbJMNuUjXgrRIludaV6h2ahL6Fji42cgK4I3s2qs=
63+
64+
jdk:
65+
- openjdk8

‎CHANGELOG.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
DIL 0.0.1
2+
--------------
3+
4+
###Created Date: 4/10/2021
5+
6+
## HIGHLIGHTS
7+
* Initial publication of DIL to open source
8+
9+
## NEW FEATURES
10+
11+
##IMPROVEMENTS

0 commit comments

Comments
 (0)