Build1 publisher3 min readPublished
AWS's Git metrics pipeline baselines AI coding tools on commit and pull request counts
AWS's walkthrough for the pattern wires GitHub and GitLab activity feeds through Step Functions into QuickSight on a schedule you set in CloudFormation, with a detector Lambda that skips the whole cycle when nothing has changed.
The Engineer · Build desk
What happened
- AWS has published a walkthrough for a serverless pipeline that pulls repository metrics from GitHub and GitLab on a schedule, stores them in Amazon S3 and charts them in Amazon QuickSight.
- A detector Lambda watches the GitHub events API and GitLab activity feeds for commits, pull requests, issues and repository creation or deletion, and processing is skipped entirely when nothing changed.
- The first execution loads all repository metadata, later runs collect only what changed, and a full refresh is triggered automatically every 24 hours to keep the data accurate.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint The fields collected are activity counts, so a pre-and-post rollout comparison cannot separate faster delivery from smaller, more numerous commits without evidence from somewhere else.
- cost Because the detector runs inside every scheduled cycle, the chosen interval sets a floor bill of one state machine execution plus one Lambda, active repositories or not.
- decision Whoever deploys it now picks the collection interval against how hard they are willing to hit the GitHub and GitLab APIs, since the incremental path exists to cut call volume.
A velocity baseline only survives a tool rollout if the thing being counted keeps its meaning. The signals AWS names are commits, pull requests, issues, and repository creation or deletion [5]. An agent that writes in many small commits lifts every one of those numbers without moving a release date. AWS makes the point itself in the post: "You won't know whether AI is speeding things up, only padding the commit counts, or quietly introducing quality issues you did not expect" [4]. For a before-and-after comparison to transfer, commit granularity and pull request size have to hold roughly constant across the two periods.
The collection path is careful work: a detector Lambda queries the GitHub events API and GitLab activity feeds, and if nothing has changed since the last collection, processing is skipped entirely [5]. That skip sits downstream of the schedule. EventBridge Scheduler starts the state machine at whatever interval you configured [11], and the state machine calls the detector before it chooses between a full and an incremental load [12]. Each cycle therefore costs at least one Step Functions execution and one Lambda invocation, pushes or no pushes [17]. At rate(1 hour) that is 24 cycles a day, one of them the mandatory full refresh and 23 candidates for the skip [18], or 168 cycles across an idle week [21].
The full refresh every 24 hours is triggered, in AWS's words, "to maintain data accuracy" [8]. Incremental collection reads an activity feed and applies deltas, so any event the feed does not surface stays missing until the next full load. A dropped event can therefore sit in the dashboard for up to a day.
Chunking is the other default to read before deploying: above 20 active repositories, Step Functions partitions them into equal-sized chunks and processes them concurrently in a Map state, one Lambda invocation per chunk; below the threshold, a single invocation collects all repositories [7]. Equal-sized means equal in repository count, not in activity, so the chunk holding your busiest monorepo sets the wall-clock time for the whole Map state. Retries use exponential backoff for transient API failures [14].
AWS calls the design one that "abstracts infrastructure management while staying low-cost at scale" [15]. It is six AWS services in a stack [10], and the first thing most teams will change is collection frequency, a CloudFormation parameter that accepts both rate() and cron() expressions [9]. That buys the extraction work AWS says has traditionally required hand-rolled ETL jobs, dedicated infrastructure and ongoing maintenance [2]. And it comes on the terms AWS set out for the AI-DLC observability pillar: "If you're using AI coding tools, you need numbers to back it up. Set a baseline, track what changes, and keep watching" [3].
What to watch
- Whether AWS extends the collected fields beyond activity counts to review or change-failure signals. Its own padding warning implies those signals are needed.
- A published running-cost example for a large repository estate at hourly collection, which would test the low-cost-at-scale claim.
- Whether the template exposes the 20-repository chunking threshold and Map state concurrency as parameters.