Skip to main content

​
Pixwel logs

​
Telemetry

We have many sources of telemetry:
  • AWS CloudWatch
  • DataDog
  • PaperTrail
  • NewRelic
Details for the above can be found in Pixwel Platform Operations and alarms talks about what alarms are available that will let us know something is wrong automatically so we can respond as fast as possible.

​
CloudWatch

We use CloudWatch to record metrics on the application. There are two types of metrics:
  1. One off event. E.g. “someone logged in”. We can average these out to determine “14 people logged in in the last 5 minutes” and derive a “LoginRate” metric. This approach will be familiar to anyone used to the StatsD/Librato type of approach.
  2. A metric. Think of this as a gauge or a meter that we can read. This is things like “number of open work requests”. See GatherKpis.php for the main bulk of the collection of metrics.
The ECS task role grants permission to write to CloudWatch, so containers can publish metrics without keypairs. Example: 
// log a counter called 'some-metric' with a prefix of KPI
MetricsLogger::log('KPI', 'some-metric', 42, 'Count');

​
How logs are shipped

ECS/Fargate services send their container logs to AWS CloudWatch Logs via the awslogs driver configured in each ecs-*.service.yml. From CloudWatch they may be forwarded on to PaperTrail / DataDog. For local development, logs go to the container’s stdout — view them with docker-compose logs -f app (and the worker services).
The legacy approach (a radify base-box Ansible role wrapping PaperTrail’s remote_syslog in an init.d service, fed by rsyslog on Ubuntu nodes) is no longer used.

​
Special case for the “magic closet”

The “magic closet” runs on OSX, not Ubuntu, is not provisioned through immutable infrastructure so any changes around logging must be made manually - see slurpee and services for more details on this setup.

​
Ingest logs

See the ingest documentation for details.
Back to docs index | Next page in recommended reading order >>