Skip to content

feat: Github App For Keploy (Beta) #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 29 commits into
base: main
Choose a base branch
from

Conversation

SkySingh04
Copy link
Contributor

@SkySingh04 SkySingh04 commented Feb 12, 2025

Fixes : keploy/keploy#2531

Keploy Github App

Streamline your code reviews with flexible, AI-powered analysis

Keploy Github App is an open-source, self-hosted AI tool designed to automate and enhance reviews of code, documentation, and more. It empowers teams to enforce standards, streamline workflows, and improve security without vendor lock-in or hidden costs. Choose any language model (LLM), including self-hosted ones via Ollama or open-source alternatives.

🌟 Key Features

1️⃣ Customizable Review Rules

  • Define your team's standards in a RULES.md, YAML, or JSON file
  • Enforce coding standards, documentation guidelines, and security best practices
  • Automatically flag inconsistencies like mixed naming conventions or inadequate error handling

2️⃣ Static Analysis + AI-Powered Review

  • Combines traditional static analysis with advanced AI reasoning
  • Analyzes PRs based on your custom rule set
  • Highlights style, security, and architectural concerns
  • Provides actionable improvement suggestions

3️⃣ Flexible LLM Integration

  • Choose the best model for your specific use case:
    • For code reviews: Models like OpenAI Codex or Mistral
    • For documentation: Models better suited for writing, such as GPT-4 or Claude
  • Supports both open-source and commercial LLMs
  • Configure via keploy-config.json file

4️⃣ GitHub Integration

  • Automatically reviews pull requests when opened or updated
  • Provides inline comments directly in your PRs
  • Works with your existing GitHub workflow

🚀 Setup Instructions

Prerequisites

  • Node.js v18 or higher
  • npm or yarn
  • GitHub account and repository (for GitHub integration)

Installation

  1. Clone the repository and checkout the branch

    git clone https://github.com/SkySingh04/testGPT.git
    git fetch
    git checkout github-app-pr
    cd keploy-github-app-beta
  2. Install dependencies

    npm install
  3. Configure the app
    Run the CLI configuration tool to set up the app for your use case:

    npm run dev

    This will prompt you to select:

    • Your use case (code review, documentation review, etc.)
    • API endpoint for your LLM
    • Preferred language model

    Alternatively, you can manually create a keploy-config.json file in the project root:

    {
      "useCase": "Code Review",
      "apiEndpoint": "YOUR_LLM_API_ENDPOINT",
      "selectedModel": "YOUR_PREFERRED_MODEL"
    }
  4. Setup GitHub App (for GitHub integration)

    • Create a new GitHub App in your GitHub account
    • Set the following permissions:
      • Pull requests: Read & Write
      • Repository contents: Read
    • Subscribe to pull request events
    • Generate and download a private key
    • Install the app on your repositories
    • Configure environment variables (create a .env file):
      APP_ID=your_github_app_id
      PRIVATE_KEY=your_github_app_private_key
      WEBHOOK_SECRET=your_webhook_secret
      
  5. Start Keploy Github App

    npm start

📋 Usage

Define Review Rules

Create a RULES.md file in your repository with your team's standards:

# Code Review Rules

## Naming Conventions
- Use camelCase for variables and functions
- Use PascalCase for classes and interfaces

## Error Handling
- All errors must be properly caught and logged
- No empty catch blocks allowed

## Documentation
- All public functions must have JSDoc comments
- Include examples for complex functions

Running Keploy Github App Locally

To review code locally before committing:

npm run start

GitHub Integration

Once configured, Keploy Github App will automatically:

  1. Monitor pull requests in repositories where the GitHub App is installed
  2. Analyze the code changes against your defined rules
  3. Provide feedback as comments directly in the PR

🔧 Advanced Configuration

Custom LLM Configuration

For using specific LLMs, update your keploy-config.json:

{
  "useCase": "Security Review",
  "apiEndpoint": "https://api.openai.com/v1/chat/completions",
  "selectedModel": "gpt-4"
}

👥 Contributing

We welcome contributions from the community! Please check our CONTRIBUTING.md guide for:

  1. Development Setup

    • Fork and clone the repository
    • Install dependencies: npm install
    • Start the development server: npm run start
  2. Coding Standards

    • Use TypeScript for all new code
    • Follow the existing code style
    • Add comments for complex logic
    • Include tests for new features
  3. Pull Request Process

    • Create a branch for your changes
    • Make your changes and commit them
    • Write clear commit messages
    • Update documentation if needed
    • Submit a PR with a description of your changes

Looking for something to work on? Check out our good first issues - these are perfect for getting started!

DEMO VIDEO :

PRism.Demo.Video.-.Made.with.Clipchamp_1740285601615.1.mp4

…tial files

Signed-off-by: Akash <akashsingh2210670@gmail.com>
Signed-off-by: Akash <akashsingh2210670@gmail.com>
Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
@SkySingh04
Copy link
Contributor Author

@Sonichigo @gouravkrosx Gentle reminder for a review!

Copy link
Member

@Hermione2408 Hermione2408 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the following comments.
Also, there is an excessive use of any. Almost all function parameters use any. Please define proper TypeScript types to ensure type safety.

import parseDiff from 'parse-diff';


async function parseGitDiffFromLLMOutput(llmOutput: any) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add type safety and null checks to avoid runtime errors instead of using any

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! Adding those all across the codebase!

export async function reviewPR(context: any, app: any, llmOutput: any) {
// export async function reviewPR(context: any, app: any) {
//trim the llmOutput to only include the diff
const ifLGTM = llmOutput.includes('LGTM');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use regex for exact match to avoid false positive

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The relevant Regex has been added!

import { reviewPR } from './diffparser.js';
import { handleLintWorkflowTrigger } from "./lint.js";

let config: any;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please avoid using any

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Their existence has been perished!

owner, repo, workflow_id: 'keploy.yaml', ref
});
} catch (error : any) {
if (error.status === 404) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 404 error handling is good, but catching only 404s and ignoring other errors can hide issues. Any specific reason for this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a debugging decision, let me refactor the error handling

};
} catch (error) {
app.log.error('Error fetching linked issue data:', error);
return null;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current implementation logs errors but does not provide a structured way to handle failure scenarios.
Consider returning proper error instead of null.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added proper error handling across the codebase!

};
}

function extractIssueNumber(text: string): number | null {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If multiple issue numbers are found in PR body/title, only the first one is processed. Is that intentional?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have enhanced linked issue handling to handle multiple linked issues, good catch!

ref
});

const content = Buffer.from(response.data.content, 'base64').toString();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If getContent fails but doesn't throw an error, response.data.content might be undefined, leading to runtime issues.
Add a null check before decoding.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Null check added!

await context.octokit.issues.createComment({
...context.repo(),
issue_number: context.payload.pull_request.number,
body: 'Failed to run security check'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of just saying "Failed to run security check," include the actual error message from GitHub API. Here also only 404 is handled any particular reason ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, this was a debugging decision, the error handling has been refactored to handle a variety of errors

await context.octokit.issues.removeLabel({
...context.repo(),
issue_number: prNumber,
name: existingLabels
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is name suppose to be an array since Object.keys will return an array

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, this was an error due to poorly handled types. This has been fixed


if (lowerResponse.includes('lgtm!')) {
return 'LGTM';
} else if (lowerResponse.includes('spam') || lowerResponse.length < 10) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic assumes shorter responses mean "Spam," but this might not always be the case. Are you sure this is a definitive check?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A simple fix for this is to just also check if the response contains keywords like

 'ok', 'yes', 'no', 'good', 'fine', 'nice', 'approved', '+1'

This change ensures that legitimate short responses won't be incorrectly labeled as spam while still maintaining the ability to catch actual spam content.

@SkySingh04
Copy link
Contributor Author

Thanks for the review @Hermione2408 , Will pick this up over the weekend!

…he application

- Added a new `types.ts` file to define TypeScript interfaces for GitHub API related entities, application context, and configuration.
- Updated function signatures in various files to use the new types, enhancing type safety and clarity.
- Refactored existing functions to replace `any` types with specific interfaces, improving maintainability and reducing potential runtime errors.
- Adjusted error handling and logging to provide clearer messages and context.

This change aims to streamline the codebase and facilitate future development by leveraging TypeScript's type system.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
… diffparser.ts and types.ts

- Introduced specific TypeScript interfaces for GitHub context and application parameters in `diffparser.ts`, improving type safety.
- Updated function signatures to replace `any` with defined types, enhancing clarity and maintainability.
- Improved error handling in `createInlineCommentsFromDiff` to provide more informative logging.
- Expanded `types.ts` with additional GitHub API parameter and response types, ensuring better integration with the GitHub API.

These changes aim to streamline the codebase and leverage TypeScript's capabilities for better development practices.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Implemented regex for improved extraction of diff blocks from LLM output, ensuring more accurate parsing.
- Updated the logic to handle cases where no valid diff is found, providing clearer error messages and logging.
- Refined the LGTM detection to use regex for exact matching, reducing false positives.

These changes aim to improve the robustness and reliability of the diff parsing functionality.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Added new methods to the GitHub context interface for listing files and review comments, improving API integration.
- Updated logging statements to use structured data for better clarity and debugging.
- Refactored code in `pr.ts` to improve type handling and error management, ensuring more robust data processing.
- Adjusted label removal logic in `addLabel.ts` to handle labels individually, enhancing reliability.

These changes aim to improve maintainability and enhance the overall robustness of the application.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
… PR management

- Enhanced error handling in `keploy.ts`, `lint.ts`, and `security.ts` to provide more specific feedback based on error status codes.
- Updated logging in `pr.ts` to capture detailed error messages and context when fetching PR files and comments.
- Improved user feedback by creating comments on PRs for various error scenarios, enhancing user experience and debugging capabilities.
- Refined error handling in `userConfig.ts` to address specific error types and provide clearer logging for configuration loading issues.

These changes aim to enhance the robustness and maintainability of the application by improving error visibility and user feedback.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
…oading

- Introduced a new `errors.ts` file to define structured error types for better error management across the application.
- Enhanced the `loadConfig` and `saveConfig` functions in `userConfig.ts` to return a `Result` type, improving error handling and logging.
- Updated the `llm.ts` and `pr.ts` files to utilize the new error handling structure, providing clearer feedback on configuration loading and linked issue data retrieval.
- Improved user feedback in the CLI by logging specific error messages when configuration loading fails.

These changes aim to enhance the robustness and maintainability of the application by providing clearer error visibility and structured handling.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Updated `index.ts` to run workflow triggers in parallel using `Promise.all`, improving performance.
- Refactored `getAllPrDetails` in `pr.ts` to execute multiple asynchronous operations concurrently, enhancing efficiency in fetching PR details.
- Modified `getLinkedIssueData` to fetch issue details and comments in parallel, streamlining data retrieval.
- Improved label removal logic in `addLabel.ts` to handle label deletions concurrently, increasing responsiveness.

These changes aim to enhance the application's performance and responsiveness by optimizing asynchronous operations across various modules.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Updated `llm.ts` to support multiple linked issues, improving the context provided in PR analysis.
- Refactored `pr.ts` to extract multiple issue numbers from PR titles and bodies, allowing for concurrent fetching of linked issues.
- Introduced a new `LinkedIssues` interface in `types.ts` to accommodate the changes in linked issue data structure.

These enhancements aim to provide a more comprehensive view of linked issues in PR analyses, improving the overall functionality of the application.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Updated the condition in `getRulesForLLM` to ensure that the response data contains valid content before processing.
- Enhanced the error message for better clarity when the content is missing or invalid.

These changes aim to provide more robust error handling and clearer feedback when loading rules from the GitHub API.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Updated the label determination logic to include a new helper function, `isValidShortResponse`, which checks for valid short responses before categorizing them as 'Spam'.
- This change improves the accuracy of label assignment based on the content of the response, ensuring that valid short responses are not incorrectly flagged.

These enhancements aim to provide more precise labeling in PR analysis, improving the overall functionality of the application.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
@SkySingh04
Copy link
Contributor Author

@Hermione2408 All of your comments have been resolved and the codebase has been wiped clean of usage of any

Also my apologies for the shit code quality, this entire thing was built in one-24hour-stretch and hence the code quality was overlooked from my end.

Thanks for the review, PTAL at the updated changes!

@SkySingh04 SkySingh04 requested a review from Hermione2408 March 15, 2025 20:49
Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Enhanced user feedback mechanisms in the PR analysis workflow, providing clearer notifications and updates during processing.
- Updated logging to capture more detailed information about the analysis steps, improving transparency for users.
- Refined error handling to ensure users receive specific feedback when issues arise, enhancing the overall user experience.

These changes aim to improve user engagement and clarity during PR analysis, making the application more user-friendly.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Removed the existing API endpoint URL in `keploy-config.json`, setting it to an empty string. This change may be part of a broader update to the configuration management or a temporary adjustment for testing purposes.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Removed the previously set model "llama2" in `keploy-config.json`, leaving the selected model as an empty string. This change may be part of a configuration reset or adjustment for future updates.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
Copy link
Member

@Hermione2408 Hermione2408 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please address the comment

);
} else {
app.log.error(
`Failed to create comment for ${filePath} line ${line}: Unknown error`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Log full error stack

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have added a function to better log the full error stack. If there is a specific reference you would prefer me to follow do let me know !

};
pulls: {
createReviewComment: (params: ReviewCommentParams) => Promise<unknown>;
listFiles: (params: { owner: string; repo: string; pull_number: number }) => Promise<{ data: any[] }>;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use Correct Type Definitions from Octokit

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have made updates to ensure we are using official type definitions from Octokit to the best of my capability. If you see a scope for improvement here, do let me know

- Removed the comment regarding GitHub API related types in `types.ts` to streamline the code and improve clarity.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Removed a comment regarding core data collection functions in `pr.ts` to improve code clarity and maintainability.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Introduced a comment at the beginning of `pr.ts` to clarify the purpose of the core data collection functions, enhancing code readability and maintainability.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Introduced three new GitHub Actions workflows: `keploy.yaml` for running tests with Keploy, `lint.yaml` for code linting across multiple languages, and `security.yaml` for performing security scans using Trivy.
- Each workflow is triggered on pull requests to the main branch and supports manual execution.
- The workflows include steps for setting up the environment, running tests or analysis, and commenting results on the pull request.

These additions enhance the CI/CD pipeline by ensuring code quality, security, and functionality before merging changes.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Modified the logging of the LLM output in `index.ts` to ensure the output is properly stringified before logging. This change enhances the clarity of the logged information during the PR analysis process.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Introduced a comprehensive README.md for the Keploy Github App, detailing its features, setup instructions, usage guidelines, and contribution process. This documentation aims to enhance user understanding and facilitate easier onboarding for new contributors.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Modified the JavaScript linting step in `lint.yaml` to use a custom ESLint configuration file located at `keploy-github-app-beta/eslint.config.js`. This change allows for more tailored linting rules and ignores specific directories.
- Added a new ESLint configuration file that specifies rules for JavaScript files, including warnings for unused variables and console statements, and enforcing semicolons.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Added the `eslint-formatter-junit` package to the JavaScript linting step in `lint.yaml`. This formatter will enhance the output of ESLint results, making it easier to integrate with CI tools and improve reporting capabilities.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Introduced a blank line at the beginning of `keploy.ts` to enhance code readability and maintain consistency with formatting standards.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Introduced a new utility function `logError` in `utils.ts` to standardize error logging with full stack traces.
- Updated various modules (`diffparser.ts`, `index.ts`, `keploy.ts`, `lint.ts`, `llm.ts`, `pr.ts`, `security.ts`, `src/addLabel.ts`, `src/config/userConfig.ts`) to utilize the new logging function, improving consistency and clarity in error reporting.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Updated `getLinkedIssueData` in `pr.ts` to handle optional fields for issue body and author, ensuring defaults are set for missing data.
- Enhanced label and assignee processing to check for array types, providing fallback values for better robustness.
- Refactored `GithubContext` in `types.ts` to utilize `ProbotOctokit` for improved type safety and clarity in GitHub API interactions.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
- Renamed the file to follow consistent casing conventions.
- Introduced `ProbotOctokit` type in the `GitHubContext` interface for improved type safety.
- Refactored the `GitHubContext` structure to include detailed payload information for better clarity in GitHub API interactions.

Signed-off-by: Sky Singh <akashsingh2210670@gmail.com>
@SkySingh04 SkySingh04 requested a review from Hermione2408 March 21, 2025 15:10
@SkySingh04
Copy link
Contributor Author

@Hermione2408 Your requested changes have been made!

Copy link

keploy bot commented Apr 15, 2025

Nice Pr 😄. We currently support Unit Test Generation only for Go projects.
Stay tuned – support for more languages is coming soon! 🚀

1 similar comment
Copy link

Nice Pr 😄. We currently support Unit Test Generation only for Go projects.
Stay tuned – support for more languages is coming soon! 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[feature]: a GitHub App to Trigger GitHub Actions
2 participants