Tailpipe's set of [plugins](https://hub.tailpipe.io/) lets you collect and query logs from a variety of sources, including [AWS](https://hub.tailpipe.io/plugins/turbot/aws), [GCP](https://hub.tailpipe.io/plugins/turbot/gcp), [Azure](https://hub.tailpipe.io/plugins/turbot/azure), and more. But what if you have logs that don't fit into any of these categories? The latest release of Tailpipe can now collect and query any log type, even if it doesn't have a plugin. This is done by creating a [custom table](https://tailpipe.io/docs/collect/custom-tables) that defines the schema for your logs. This unlocks the ability to use the power of Tailpipe for pretty much any log type you can think of. ## Configure Tailpipe to collect custom logs For a custom log, you need to define a [partition](https://tailpipe.io/docs/reference/config-files/partition), [format](https://tailpipe.io/docs/reference/config-files/format), and [table](https://tailpipe.io/docs/reference/config-files/table). Tailpipe has several built-in formats, and formats can also be provided by plugins. This example uses the `jsonl` format, which is one of the core formats (others include `grok`, `regex`, and `delimited`). While Tailpipe already has a [plugin for GitHub audit logs](https://hub.tailpipe.io/plugins/turbot/github), we can also use the custom feature to have more flexibility (and to demonstrate this feature). ```hcl partition "my_github" "demo" { source "file" { paths = ["/Users/mattstratton/logs_tailpipe/custom"] file_layout = `%{DATA}.json.gz` } } table "my_github" { format = format.jsonl.default column "tp_timestamp" { transform = `to_timestamp("@timestamp" / 1000)` } } ``` The above table definition uses the `jsonl.default` format preset, which means we don't have to tell Tailpipe anything about the structure of the logs - use of the `default` preset tells Tailpipe to expect a simple JSON object per line. We do need to tell Tailpipe where to find the timestamp for our logs, however. In this case, the timestamp of our example logs stores the timestamp as Unix epoch time in milliseconds. We use the `transform` function to convert this to a standard DuckDB timestamp format. Take note also of the use of backticks (\`) around the `transform` function. This is a new feature in Tailpipe that allows us to use backticks instead of escaping quotes, making the configuration cleaner and easier to read. That's all we need to start collecting logs! ## Collecting the custom logs We collect our logs with `tailpipe collect`, just like any other logs. ```shell $ tailpipe collect my_github.demo --from 2024-04-01 Collecting logs for my_github.demo from 2024-04-01 Artifacts: Discovered: 2 Downloaded: 2 1.3MB Extracted: 2 Rows: Received: 21,718 Enriched: 21,718 Saved: 21,074 Filtered: 644 Files: Compacted: 676 => 345 Completed: 4s ``` If we select from this table, we can see there are a lot of columns: ```shell $ tailpipe query "select * from my_github limit 5;" +---------------+------------------------+----------------------+--------+-------------------+----------+--------------+----------------+----------------+--------+----------+--------------+-----------------------------------+--------------------------------------+-----------------------+------------------------+--------+--------+----------+-------------+------------+--------+------------------+---------------+-------------------------------+-------------------+--------+--------+---------+----------------------------------------------+-------------+----------+---------+------------------------------------+-------------+---------------+---------+---------+----------------------------------------------+-----------------------------+-------------------------------+-------------------------------+------------+--------+--------+----------------------+----------+----------------+----------------+------------+---------+------------------+--------+------------+------------+--------+-----------------------------------+------------+--------------+--------------+----------------+-------------+-----------------+-----------------+----------------------------------------+------------------------------------------------------------------------+---------------------------------------------------------+--------+---------+----------+---------------------------+----------+--------------------+--------------------------+----------------------+----------------------------+---------------+----------------------+--------------------------------+------------------+---------------------------+----------------------------+---------------------------------+----------------------------------------+-----------------------------------------------------+------------------------------------------+-----------+---------------+------------+-------------+--------------------------+-----------------------------------------+------------+--------------------------------------+--------+--------+-------------+--------------+--------+------------+-----------+---------------------------------------------------------------------------------------------------------------+----------+------------+-----------------------------+-------------------------------+-------------+-----------------+---------+---------------------+-------------------+------------+-----------+--------------------------------------+----------+---------------------+--------+--------------+--------------+--------------------+----------------+----------------+-----------+---------+---------------------+--------------+ | @timestamp | _document_id | action | active | actor | actor_id | actor_is_bot | actor_location | admin_enforced | after | alert_id | alert_number | allow_deletions_enforcement_level | allow_force_pushes_enforcement_level | application_client_id | authorized_actor_names | before | branch | business | business_id | comment_id | config | create_protected | created_at | dismiss_stale_reviews_on_push | enforcement_level | event | events | ghsa_id | hashed_token | head_branch | head_sha | hook_id | ignore_approvals_from_contributors | integration | invitation_id | invitee | inviter | linear_history_requirement_enforcement_level | lock_allows_fetch_and_merge | lock_branch_enforcement_level | merge_queue_enforcement_level | multi_repo | name | number | oauth_application_id | old_name | old_permission | operation_type | org | org_id | overridden_codes | owner | owner_type | permission | plan | programmatic_access_type | project_id | project_kind | project_name | public_project | public_repo | publicly_leaked | pull_request_id | pull_request_reviews_enforcement_level | pull_request_title | pull_request_url | reason | reasons | referrer | repo | repo_id | repositories_added | repositories_added_names | repositories_removed | repositories_removed_names | repository_id | repository_selection | request_access_security_header | request_category | require_code_owner_review | require_last_push_approval | required_approving_review_count | required_deployments_enforcement_level | required_review_thread_resolution_enforcement_level | required_status_checks_enforcement_level | review_id | reviewer_type | run_number | secret_type | secret_type_display_name | signature_requirement_enforcement_level | started_at | strict_required_status_checks_policy | team | title | token_id | token_scopes | topic | trigger_id | user | user_agent | user_id | visibility | vulnerability_alert_rule_id | vulnerability_alert_rule_name | workflow_id | workflow_run_id | tp_akas | tp_date | tp_destination_ip | tp_domains | tp_emails | tp_id | tp_index | tp_ingest_timestamp | tp_ips | tp_partition | tp_source_ip | tp_source_location | tp_source_name | tp_source_type | tp_table | tp_tags | tp_timestamp | tp_usernames | +---------------+------------------------+----------------------+--------+-------------------+----------+--------------+----------------+----------------+--------+----------+--------------+-----------------------------------+--------------------------------------+-----------------------+------------------------+--------+--------+----------+-------------+------------+--------+------------------+---------------+-------------------------------+-------------------+--------+--------+---------+----------------------------------------------+-------------+----------+---------+------------------------------------+-------------+---------------+---------+---------+----------------------------------------------+-----------------------------+-------------------------------+-------------------------------+------------+--------+--------+----------------------+----------+----------------+----------------+------------+---------+------------------+--------+------------+------------+--------+-----------------------------------+------------+--------------+--------------+----------------+-------------+-----------------+-----------------+----------------------------------------+------------------------------------------------------------------------+---------------------------------------------------------+--------+---------+----------+---------------------------+----------+--------------------+--------------------------+----------------------+----------------------------+---------------+----------------------+--------------------------------+------------------+---------------------------+----------------------------+---------------------------------+----------------------------------------+-----------------------------------------------------+------------------------------------------+-----------+---------------+------------+-------------+--------------------------+-----------------------------------------+------------+--------------------------------------+--------+--------+-------------+--------------+--------+------------+-----------+---------------------------------------------------------------------------------------------------------------+----------+------------+-----------------------------+-------------------------------+-------------+-----------------+---------+---------------------+-------------------+------------+-----------+--------------------------------------+----------+---------------------+--------+--------------+--------------+--------------------+----------------+----------------+-----------+---------+---------------------+--------------+ | 1710905872881 | isf3hi6IKXPncM45PN1KhA | issue_comment.update | | netlify[bot] | 40209326 | true | | | | | | | | | | | | | | | | | 1710905872881 | | | | | | ro7FUz99uqrzSOr7spjAo/aBDYWPHWfoDoRSh6ouwCo= | | | | | | | | | | | | | | | | | | | modify | devopsdays | 2938293 | | | | | | GitHub App server-to-server token | | | | | true | | | | | | | | | devopsdays/devopsdays-web | 45844314 | | | | | | | | | | | | | | | | | | | | | | | | | 85582235393 | | | | | Octokit Ruby Gem 4.22.0 | | | | | | | | 2024-03-19 00:00:00 | | | | 0dcd7918-6cec-4a35-9a6b-d449de932116 | default | 2025-04-14 23:45:12 | | demo | | | | | my_github | | 2024-03-20 03:37:52 | | | 1710905763388 | DK8s-uWa6n5yGL7e-JE07Q | issue_comment.update | | netlify[bot] | 40209326 | true | | | | | | | | | | | | | | | | | 1710905763388 | | | | | | ro7FUz99uqrzSOr7spjAo/aBDYWPHWfoDoRSh6ouwCo= | | | | | | | | | | | | | | | | | | | modify | devopsdays | 2938293 | | | | | | GitHub App server-to-server token | | | | | true | | | | | | | | | devopsdays/devopsdays-web | 45844314 | | | | | | | | | | | | | | | | | | | | | | | | | 85582235393 | | | | | Octokit Ruby Gem 4.22.0 | | | | | | | | 2024-03-19 00:00:00 | | | | c4cf4071-3648-408f-8fda-b682c77d0d5c | default | 2025-04-14 23:45:12 | | demo | | | | | my_github | | 2024-03-20 03:36:03 | | | 1710902220282 | BVHf7Sh57OnUc1lWhUW9UQ | issue_comment.update | | netlify[bot] | 40209326 | true | | | | | | | | | | | | | | | | | 1710902220282 | | | | | | mYeGZ0CV9mayrnILSentfVXziqQUddTqmWnNdHmjn38= | | | | | | | | | | | | | | | | | | | modify | devopsdays | 2938293 | | | | | | GitHub App server-to-server token | | | | | true | | | | | | | | | devopsdays/devopsdays-web | 45844314 | | | | | | | | | | | | | | | | | | | | | | | | | 85574914084 | | | | | Octokit Ruby Gem 4.22.0 | | | | | | | | 2024-03-19 00:00:00 | | | | 283bfacc-5176-430f-a949-e51665bff519 | default | 2025-04-14 23:45:12 | | demo | | | | | my_github | | 2024-03-20 02:37:00 | | | 1710902079531 | -jBgkLRhNJcJ6snU4Q1Jrw | issue_comment.update | | unfurl-links[bot] | 33112876 | true | | | | | | | | | | | | | | | | | 1710902079531 | | | | | | WwELTHjStmlKP70VtHH5R2k4QZxYzwQ1UecRrgwONFU= | | | | | | | | | | | | | | | | | | | modify | devopsdays | 2938293 | | | | | | GitHub App server-to-server token | | | | | true | | | | | | | | | devopsdays/devopsdays-web | 45844314 | | | | | | | | | | | | | | | | | | | | | | | | | 85577577999 | | | | | octokit.js/15.18.3 Node.js/12.22.11 (Linux 5.15; x64) | | | | | | | | 2024-03-19 00:00:00 | | |