Empty Body Sent to HTTP Subscribers

[sgtoj]

Sending messages to http subscribers do not work. A POST request is made to the endpoint, but no data is sent in the body. However, the content-length header is set the expected value as if a message was sent. I got the same result for two different http servers: netcat and server.js (nodejs express wrapper). Additionally, I confirmed messages sent to the same topic but different types of subscribers (i.e. file protocol) does work as expected.

command sent

aws sns --endpoint-url http://localhost:9911 publish --topic-arn arn:aws:sns:us-east-1:1465414804035:test-topic --message "test"

verbose logging of received msg from sns

POST /message HTTP/1.1
MessageExchangeId: 9a874dc5-3dc5-477f-90d2-4c4dd6e77c35
x-amz-sns-message-id: 9a874dc5-3dc5-477f-90d2-4c4dd6e77c35
x-amz-sns-message-type: Notification
x-amz-sns-subscription-arn: 6df4ed2b-a650-4f7c-910a-1a89c7cae5a6
x-amz-sns-topic-arn: arn:aws:sns:us-east-1:1465414804035:test-topic
User-Agent: Jakarta Commons-HttpClient/3.1
Host: subscriber:8010
Content-Length: 91

# no body data?

environment setup

./docker-compose.yml

version: "3.4"
services:
  sns:
    image: s12v/sns
    environment:
    - SUBSCRIBER_HTTP=subscriber:8010
    ports:
    - 9911:9911
    volumes:
    - ./sns:/etc/sns
  subscriber:
    image: subfuzion/netcat
    ports:
    - 8010:8010
    command: "-vl 8010"

./sns/db.json

{
    "version": 1,
    "timestamp": 1465414804110,
    "subscriptions": [
        {
            "arn": "6df4ed2b-a650-4f7c-910a-1a89c7cae5a6",
            "topicArn": "arn:aws:sns:us-east-1:1465414804035:test-topic",
            "endpoint": "http:{{env:SUBSCRIBER_HTTP}}/message",
            "owner": "",
            "protocol": "http"
        }
    ],
    "topics": [
        {
            "arn": "arn:aws:sns:us-east-1:1465414804035:test-topic",
            "name": "tms-session-api-events"
        }
    ]
}

[kfelichko]

I was able to work around this by setting the content-type of the request on the receiving end, which is missing from the SNS publish.

const app = express();
app.use((req, res, next) => {
      req.headers['content-type'] = req.headers['content-type'] || 'application/json';
      next();
});
app.use(bodyParser.json());

HTH.

[victorsferreira]

in my case I just used bodyParser.text()

because it was sending a json content with a content type "plain/text". i also had to parse the json by myself

[damionvega]

Was banging my head on my keyboard for a while on this. Posting for anyone using Express v4+:

app.use(
  express.json({
    type: [
      'application/json',
      'text/plain', // AWS sends this content-type for its messages/notifications
    ],
  })
)

[alannesta]

I was able to work around this by setting the content-type of the request on the receiving end, which is missing from the SNS publish.

const app = express();
app.use((req, res, next) => {
      req.headers['content-type'] = req.headers['content-type'] || 'application/json';
      next();
});
app.use(bodyParser.json());

HTH.

I was able to work around this by setting the content-type of the request on the receiving end, which is missing from the SNS publish.

const app = express();
app.use((req, res, next) => {
      req.headers['content-type'] = req.headers['content-type'] || 'application/json';
      next();
});
app.use(bodyParser.json());

HTH.

The content type is text/plain, so req.headers['content-type'] = req.headers['content-type'] || 'application/json'; will not work.

x-forwarded-proto': 'https',
 | 2020-03-06 04:24:43 +00:00:      'cf-visitor': '{"scheme":"https"}',
 | 2020-03-06 04:24:43 +00:00:      'x-amz-sns-message-type': 'SubscriptionConfirmation',
  | 2020-03-06 04:24:43 +00:00:      'x-amz-sns-message-id': 'acdcb964-a042-434f-9746-036713666477',
 | 2020-03-06 04:24:43 +00:00:      'x-amz-sns-topic-arn': 'arn:aws:sns:us-east-1:722522486928:xxxx',
 | 2020-03-06 04:24:43 +00:00:      'content-type': 'text/plain; charset=UTF-8',
 | 2020-03-06 04:24:43 +00:00:      'user-agent': 'Amazon Simple Notification Service Agent',

[diegofcornejo]

Was banging my head on my keyboard for a while on this. Posting for anyone using Express v4+:

app.use(
  express.json({
    type: [
      'application/json',
      'text/plain', // AWS sends this content-type for its messages/notifications
    ],
  })
)

This worked for me, thanks

[geekhunger]

Solution from @damionvega still applies in January 2022! - However, I didn't want to always parse every "text/plain" request as JSON. Instead I wanted this workaround to only happen for Amazon Services (like SES or SNS). I figured that I could simply use the "user-agent" to identify these requests:

router.use(function(req, res, nxt) {
    const json_parser_options = {
        type: ["json", "*+json", "*/json", "application/json"],
        strict: true,
        limit: "5mb"
    }
    if(/Amazon/i.test(req.header("user-agent"))) {
        json_parser_options.type.push("text/*")
    }
    express.json(json_parser_options)(req, res, nxt)
})