Export a single file for each disruption - #1419
Conversation
Example JSON from this API
{
"data": {
"attributes": {
"description": "GL closed west of North Station, except Babcock-BC and Kenmore-Riverside. B and C shuttles to Copley and Back Bay. \n\nExtra OL service. Mon-Th and Friday services activated here.",
"hastus_exports": [
"https://s3.amazonaws.com/mbta-arrow/dev/local/ecfleischman/hastus-export-uploads/1785784414_1785266462_2026-summer-gl-aug8-16_disruption_82_disruption_1.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAWKDUC67O3JECNUFK%2F20260804%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260804T180533Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=1230f6ce4b1301b8e88c8cac9fca53aaee8a866d3147df139e9165ff57311b94"
],
"mode": "subway",
"status": "pending",
"title": "Central Subway surge, 8/8 - 8/16/26",
"trainsformer_exports": []
},
"id": "1",
"relationships": {
"limits": {
"data": []
},
"replacement_services": {
"data": [
{
"id": "1",
"type": "replacement_service"
}
]
},
"shuttles": {
"data": [
{
"id": "1",
"type": "shuttle"
}
]
}
},
"type": "disruption_v2"
},
"included": [
{
"attributes": {
"destination": "Babcock St",
"direction_desc": "Outbound",
"direction_id": "0",
"shape_id": "BackBayToBabcock-S",
"waypoint": "Copley"
},
"id": "1",
"relationships": {
"route_stops": {
"data": [
{
"id": "1",
"type": "shuttle_route_stop"
},
{
"id": "2",
"type": "shuttle_route_stop"
},
{
"id": "3",
"type": "shuttle_route_stop"
},
{
"id": "4",
"type": "shuttle_route_stop"
},
{
"id": "5",
"type": "shuttle_route_stop"
},
{
"id": "6",
"type": "shuttle_route_stop"
},
{
"id": "7",
"type": "shuttle_route_stop"
},
{
"id": "8",
"type": "shuttle_route_stop"
}
]
}
},
"type": "shuttle_route"
},
{
"attributes": {
"direction_id": "1",
"display_stop_id": null,
"stop_sequence": 1,
"time_to_next_stop": "142"
},
"id": "9",
"relationships": {
"gtfs_stop": {
"data": {
"id": "934",
"type": "gtfs_stop"
}
},
"stop": {
"data": null
}
},
"type": "shuttle_route_stop"
},
{
"attributes": {
"lat": 42.350162,
"lon": -71.10663,
"name": "Commonwealth Ave @ St Mary's St"
},
"id": "953",
"type": "gtfs_stop"
},
{
"attributes": {
"direction_id": "1",
"display_stop_id": null,
"stop_sequence": 6,
"time_to_next_stop": "256"
},
"id": "14",
"relationships": {
"gtfs_stop": {
"data": null
},
"stop": {
"data": {
"id": "215",
"type": "stops"
}
}
},
"type": "shuttle_route_stop"
},
{
"attributes": {
"lat": 42.349788,
"lon": -71.106391,
"name": "Commonwealth Ave @ St Mary's St"
},
"id": "938",
"type": "gtfs_stop"
},
{
"attributes": {
"direction_id": "0",
"display_stop_id": null,
"stop_sequence": 5,
"time_to_next_stop": "30"
},
"id": "5",
"relationships": {
"gtfs_stop": {
"data": {
"id": "952",
"type": "gtfs_stop"
}
},
"stop": {
"data": null
}
},
"type": "shuttle_route_stop"
},
{
"attributes": {
"lat": 42.34897,
"lon": -71.099643,
"name": "Commonwealth Ave @ Blandford St"
},
"id": "941",
"type": "gtfs_stop"
},
{
"attributes": {
"lat": 42.349397,
"lon": -71.103055,
"name": "Commonwealth Ave @ Hinsdale St"
},
"id": "939",
"type": "gtfs_stop"
},
{
"attributes": {
"direction_id": "0",
"display_stop_id": null,
"stop_sequence": 3,
"time_to_next_stop": "71"
},
"id": "3",
"relationships": {
"gtfs_stop": {
"data": null
},
"stop": {
"data": {
"id": "214",
"type": "stops"
}
}
},
"type": "shuttle_route_stop"
},
{
"attributes": {
"lat": 42.348245,
"lon": -71.076218,
"name": "Stuart St @ Dartmouth St"
},
"id": "71855",
"type": "gtfs_stop"
},
{
"attributes": {
"lat": 42.349458,
"lon": -71.10074,
"name": "Commonwealth Ave @ Silber Way"
},
"id": "951",
"type": "gtfs_stop"
},
{
"attributes": {
"end_date": "2026-08-16",
"reason": null,
"start_date": "2026-08-08",
"timetable": null
},
"id": "1",
"relationships": {
"disruption": {
"data": {
"id": "1",
"type": "disruption_v2"
}
},
"shuttle": {
"data": {
"id": "1",
"type": "shuttle"
}
}
},
"type": "replacement_service"
},
{
"attributes": {
"lat": 42.35131,
"lon": -71.115933,
"name": "Commonwealth Ave @ Buick St"
},
"id": "956",
"type": "gtfs_stop"
},
{
"attributes": {
"at_street": "Kenmore Street",
"inserted_at": "2026-08-03T18:57:47Z",
"municipality": "Boston",
"on_street": "Commonwealth Avenue",
"platform_name": "Westbound Shuttle",
"stop_desc": "Kenmore - Green Line Westbound Shuttle - Commonwealth Ave @ Kenmore St",
"stop_id": "9170151",
"stop_lat": 42.34916,
"stop_lon": -71.0949,
"stop_name": "Kenmore - Commonwealth Ave @ Kenmore St",
"updated_at": "2026-08-03T18:57:47Z"
},
"id": "214",
"type": "stops"
},
{
"attributes": {
"at_street": "Massachusetts Avenue",
"inserted_at": "2026-08-03T18:57:47Z",
"municipality": "Boston",
"on_street": "Commonwealth Avenue",
"platform_name": "Westbound Shuttle",
"stop_desc": "Hynes - Green Line Westbound Shuttle - Commonwealth Ave @ Massachusetts Ave",
"stop_id": "9170153",
"stop_lat": 42.34937,
"stop_lon": -71.088504,
"stop_name": "Hynes - Commonwealth Ave @ Massachusetts Ave",
"updated_at": "2026-08-03T18:57:47Z"
},
"id": "216",
"type": "stops"
},
{
"attributes": {
"at_street": "Massachusetts Avenue",
"inserted_at": "2026-08-03T18:57:47Z",
"municipality": "Boston",
"on_street": "Commonwealth Avenue",
"platform_name": "Park Street & North Shuttle",
"stop_desc": "Hynes - Green Line Park Street & North Shuttle - Commonwealth Ave @ Massachusetts Ave",
"stop_id": "9170152",
"stop_lat": 42.34889,
"stop_lon": -71.08835,
"stop_name": "Hynes - Commonwealth Ave @ Massachusetts Ave",
"updated_at": "2026-08-03T18:57:47Z"
},
"id": "215",
"type": "stops"
},
{
"attributes": {
"direction_id": "0",
"display_stop_id": null,
"stop_sequence": 4,
"time_to_next_stop": "31"
},
"id": "4",
"relationships": {
"gtfs_stop": {
"data": {
"id": "951",
"type": "gtfs_stop"
}
},
"stop": {
"data": null
}
},
"type": "shuttle_route_stop"
},
{
"attributes": {
"direction_id": "0",
"display_stop_id": null,
"stop_sequence": 8,
"time_to_next_stop": null
},
"id": "8",
"relationships": {
"gtfs_stop": {
"data": {
"id": "933",
"type": "gtfs_stop"
}
},
"stop": {
"data": null
}
},
"type": "shuttle_route_stop"
},
{
"attributes": {
"lat": 42.349831,
"lon": -71.103801,
"name": "Commonwealth Ave @ Granby St"
},
"id": "952",
"type": "gtfs_stop"
},
{
"attributes": {
"direction_id": "1",
"display_stop_id": null,
"stop_sequence": 3,
"time_to_next_stop": "36"
},
"id": "11",
"relationships": {
"gtfs_stop": {
"data": {
"id": "939",
"type": "gtfs_stop"
}
},
"stop": {
"data": null
}
},
"type": "shuttle_route_stop"
},
{
"attributes": {
"lat": 42.35123,
"lon": -71.118201,
"name": "Commonwealth Ave @ Pleasant St"
},
"id": "934",
"type": "gtfs_stop"
},
{
"attributes": {
"disrupted_route_id": "Green-B",
"shuttle_name": "BabcockStToBackBayviaCopley-S",
"status": "active",
"suffix": null
},
"id": "1",
"relationships": {
"routes": {
"data": [
{
"id": "1",
"type": "shuttle_route"
},
{
"id": "2",
"type": "shuttle_route"
}
]
}
},
"type": "shuttle"
},
{
"attributes": {
"direction_id": "0",
"display_stop_id": null,
"stop_sequence": 1,
"time_to_next_stop": "196"
},
"id": "1",
"relationships": {
"gtfs_stop": {
"data": {
"id": "71855",
"type": "gtfs_stop"
}
},
"stop": {
"data": null
}
},
"type": "shuttle_route_stop"
},
{
"attributes": {
"direction_id": "1",
"display_stop_id": null,
"stop_sequence": 7,
"time_to_next_stop": null
},
"id": "15",
"relationships": {
"gtfs_stop": {
"data": {
"id": "71855",
"type": "gtfs_stop"
}
},
"stop": {
"data": null
}
},
"type": "shuttle_route_stop"
},
{
"attributes": {
"direction_id": "0",
"display_stop_id": null,
"stop_sequence": 2,
"time_to_next_stop": "109"
},
"id": "2",
"relationships": {
"gtfs_stop": {
"data": null
},
"stop": {
"data": {
"id": "216",
"type": "stops"
}
}
},
"type": "shuttle_route_stop"
},
{
"attributes": {
"destination": "Back Bay",
"direction_desc": "Inbound",
"direction_id": "1",
"shape_id": "BabcockToBackBay-S",
"waypoint": "Copley"
},
"id": "2",
"relationships": {
"route_stops": {
"data": [
{
"id": "9",
"type": "shuttle_route_stop"
},
{
"id": "10",
"type": "shuttle_route_stop"
},
{
"id": "11",
"type": "shuttle_route_stop"
},
{
"id": "12",
"type": "shuttle_route_stop"
},
{
"id": "13",
"type": "shuttle_route_stop"
},
{
"id": "14",
"type": "shuttle_route_stop"
},
{
"id": "15",
"type": "shuttle_route_stop"
}
]
}
},
"type": "shuttle_route"
},
{
"attributes": {
"direction_id": "1",
"display_stop_id": null,
"stop_sequence": 4,
"time_to_next_stop": "59"
},
"id": "12",
"relationships": {
"gtfs_stop": {
"data": {
"id": "941",
"type": "gtfs_stop"
}
},
"stop": {
"data": null
}
},
"type": "shuttle_route_stop"
},
{
"attributes": {
"direction_id": "1",
"display_stop_id": null,
"stop_sequence": 5,
"time_to_next_stop": "90"
},
"id": "13",
"relationships": {
"gtfs_stop": {
"data": null
},
"stop": {
"data": {
"id": "110",
"type": "stops"
}
}
},
"type": "shuttle_route_stop"
},
{
"attributes": {
"at_street": "Kenmore Street",
"inserted_at": "2026-08-03T18:57:47Z",
"municipality": "Boston",
"on_street": "Commonwealth Avenue",
"platform_name": "Green Line Shuttle",
"stop_desc": "Kenmore - Green Line Shuttle - 512 Commonwealth Ave",
"stop_id": "9070150",
"stop_lat": 42.348712,
"stop_lon": -71.095526,
"stop_name": "Kenmore - 512 Commonwealth Ave",
"updated_at": "2026-08-03T18:57:47Z"
},
"id": "110",
"type": "stops"
},
{
"attributes": {
"lat": 42.351568,
"lon": -71.121168,
"name": "Commonwealth Ave @ Babcock St"
},
"id": "933",
"type": "gtfs_stop"
},
{
"attributes": {
"direction_id": "0",
"display_stop_id": null,
"stop_sequence": 7,
"time_to_next_stop": "59"
},
"id": "7",
"relationships": {
"gtfs_stop": {
"data": {
"id": "956",
"type": "gtfs_stop"
}
},
"stop": {
"data": null
}
},
"type": "shuttle_route_stop"
},
{
"attributes": {
"direction_id": "0",
"display_stop_id": null,
"stop_sequence": 6,
"time_to_next_stop": "107"
},
"id": "6",
"relationships": {
"gtfs_stop": {
"data": {
"id": "953",
"type": "gtfs_stop"
}
},
"stop": {
"data": null
}
},
"type": "shuttle_route_stop"
},
{
"attributes": {
"direction_id": "1",
"display_stop_id": null,
"stop_sequence": 2,
"time_to_next_stop": "33"
},
"id": "10",
"relationships": {
"gtfs_stop": {
"data": {
"id": "938",
"type": "gtfs_stop"
}
},
"stop": {
"data": null
}
},
"type": "shuttle_route_stop"
}
],
"jsonapi": {
"version": "1.0"
}
} |
|
Oops, I just realized I forgot the UI component of this ticket |
|
I am no frontend dev, so I'd appreciate any feedback on better UX/design for the download button. |
|
I can't deploy your change to a lower environment: Maybe you need to rebase off of master to get @lemald 's changes to the deploy process? |
|
Actually, hold on a bit before testing, I'm realizing I have some issues in the query so it's not returning all the necessary data. |
| alias Plug.Conn | ||
|
|
||
| @spec index(Conn.t(), map()) :: Conn.t() | ||
| def index(conn, %{"id" => id}) do |
There was a problem hiding this comment.
issue: running locally, I noticed this endpoint crashes if the disruption does not exist. it should return a 404 instead of crashing.
There was a problem hiding this comment.
I think that is expected behavior when running locally — if you set debug_errors: false in config/dev.exs, it will return a 404. You can also see this behavior with the frontend /disruption/:id endpoint, where it shows a stack trace when running locally but a 404 in deployed environments.
There was a problem hiding this comment.
The new endpoint does give a 400 now when I supply a nonexistent ID... this is better than crashing, but I would still prefer the more descriptive 404
This is only a problem if we need to make changes to a disruption that we've already included in gtfs inputs, right? Anyway, this is the same logic as the existing I think changing it to return an S3 URI is fine, but it will require changes in the corresponding parts of gtfs creator ( |
I think the joins were just making things more difficult to read, and I don't think there's a benefit to doing them in a single query rather than in separate queries in the preload.
|
issue: endpoint should include S3 URI for the shape instead of the shape ID. |
|
I just realized that I missed a usage of the arrow API, in Or alternatively, the |
|
@zyxw59 I think we would want to include the other information from the service schedule APIs in addition to the S3 URI - this information tells |
|
Augh, after drilling down into all the things we need this API to produce, I think I'm appreciating your distaste for JSON API 🙃 |
|
I think I'm going to majorly refactor this away from JSON API |
|
Hmm, I keep discovering more places in GTFS Creator that are calling the Arrow API. Specifically |
|
no worries! I think that's totally acceptable, and I had a similar realization on a previous arrow PR I did |
|
And one more usage that isn't directly tied to specific disruptions: The |
|
@rudiejd how do you think we should handle those uses that aren't tied to a specific disruption (archived shuttle routes and merge replacement bus stops)? |
|
Actually, as I think about this a bit more, an Arrow In particular, I'm imagining a scenario where there are two disruptions that use the same shuttle, and then there is a change to the shuttle but only one of the disruption exports is updated, so now there are two shuttles with the same IDs but different definitions. |
|
Yeah, I probably hadn't thought through this completely... only the activation of the shuttle is really tied to the disruption. I think one approach could be to handle the shapes, archived shuttles, and all of the shuttle data separately. what would you think about storing the shuttle as a separate entity in S3, and then having the disruption API reference the S3 URI of that entity? |
|
The alternative (and maybe a simpler approach) to that would be that we consider each disruption to include its own snapshot of the shuttles, so we just include all of the shuttle information on this endpoint regardless of whether it might be different in a different disruption. This is more along the lines of what I was thinking |
I think the issue there is that if two disruptions have a different snapshot of the same shuttle, then GTFS creator will try to create two routes with the same ID but different definitions. |
I don't think we need/want to store the shuttles in S3. I think the shuttle definitions could just be JSON files downloaded from Arrow and stored in the GTFS inputs repo. |
ahhhhhh yeah. we could have some logic to avoid a collision in |
yeah that makes more sense. Ignore the S3 part, I think I might have been too focused on what we're doing in this PR. But same idea. In this case, you could just reference the shuttle by name in the disruption API. Or we could use Arrow's auto-increment ID as a unique identifier for the shuttles if you think that's better. |
| ], | ||
| limits: [:limit_day_of_weeks], | ||
| replacement_services: [ | ||
| shuttle: [routes: [:shape, route_stops: [:stop]]] |
There was a problem hiding this comment.
if we're not using this preload, could you remove it? namely, I think only the shuttle name should matter
There was a problem hiding this comment.
No, it's needed for getting the timetables for the replacement service.
Though we can get rid of shape.
There was a problem hiding this comment.
But maybe we don't want to be including the timetables in this export? They are quite large (for the Green Line disruption I'm using for testing, ~378 kB, vs >2 kB for all the rest of the export data).
There was a problem hiding this comment.
Maybe we could upload the timetables to S3 as well?
There was a problem hiding this comment.
hmm, I think we should be fine since this is probably one of the bigger disruptions and it's still well under the github file limit. This is about the size of the Southside base schedules, which we already store multiple copies of in this repo with no noticable problems in clone / commit speed
There was a problem hiding this comment.
I think it's really only when we get files on the order of 10MB+ that I start to notice a problem
There was a problem hiding this comment.
I guess there is also the question of whether GitHub will show diffs for files that big, but we might just not care that much about that (ideally disruption exports are not frequently modified once they've been created)
3 participants
Summary of changes
Asana Ticket: 🏹Arrow produces single file describing a disruption
Adds a new API endpoint,
/api/disruption/:id, which returns a JSON object containing all the data needed by GTFS creator related to that disruption.Reviewer Checklist