<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Taleshape | Blog</title><description/><link>https://taleshape.com/</link><language>en</language><item><title>Build Your Own Data Analytics Slack Agent With Shaper</title><link>https://taleshape.com/blog/build-your-own-data-analytics-slack-ai-agent-with-shaper/</link><guid isPermaLink="true">https://taleshape.com/blog/build-your-own-data-analytics-slack-ai-agent-with-shaper/</guid><pubDate>Wed, 15 Apr 2026 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;At Taleshape, we are building &lt;a href=&quot;https://taleshape.com/shaper/docs&quot;&gt;Shaper&lt;/a&gt; - an &lt;strong&gt;open source SQL-based tool to visualize data and build dashboards&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;And we think Shaper is a great fit for building custom data analytics AI agents.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Let’s look at an agent in action:&lt;/strong&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Talk to the agent &lt;strong&gt;directly in Slack&lt;/strong&gt;, and it uses Shaper to answer your data questions.&lt;/li&gt;
&lt;li&gt;The agent can create anything from &lt;strong&gt;charts&lt;/strong&gt; and &lt;strong&gt;Excel&lt;/strong&gt; files to complete &lt;strong&gt;PDF reports&lt;/strong&gt;, and then shares these files in Slack.&lt;/li&gt;
&lt;li&gt;Since any chart or report in Shaper is just a SQL query, we can easily &lt;strong&gt;verify and reproduce&lt;/strong&gt; any answer the agent gives.&lt;/li&gt;
&lt;/ul&gt;
&lt;lite-youtube videoid=&quot;evkRKs-M0Lc&quot;&gt; &lt;a href=&quot;https://youtube.com/watch?v=evkRKs-M0Lc&quot;&gt; &lt;span&gt;Play&lt;/span&gt; &lt;/a&gt; &lt;/lite-youtube&gt;  
&lt;div&gt;&lt;h2 id=&quot;how-it-works&quot;&gt;How It Works&lt;/h2&gt;&lt;/div&gt;
&lt;ol&gt;
&lt;li&gt;The agent itself is a simple Python application that listens to Slack messages.&lt;/li&gt;
&lt;li&gt;When it receives a message, it uses an LLM to answer the user request.&lt;/li&gt;
&lt;li&gt;We give the LLM tools to use &lt;a href=&quot;https://taleshape.com/shaper/docs/api-endpoints/&quot;&gt;Shaper’s API&lt;/a&gt; to generate png/pdf/xlsx files directly from SQL queries.&lt;/li&gt;
&lt;li&gt;We then upload the generated files to Slack. While the LLM generates the SQL, it doesn’t touch the results, avoiding the risk of hallucinated data or charts.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;&lt;img src=&quot;https://taleshape.com/_astro/slack-agent-architecture.CJgznD_5_2toyiE.svg&quot; alt=&quot;Architecture Diagram&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; fetchpriority=&quot;auto&quot; width=&quot;800&quot; height=&quot;400&quot;&gt;&lt;/p&gt;
&lt;p&gt;Since we control the agent and Shaper runs in our own infrastructure, we are also in &lt;strong&gt;full control of what data the LLM sees&lt;/strong&gt; and which LLM we use.
This setup can be easily adapted to fit the privacy and security needs of any organization.&lt;/p&gt;
&lt;div&gt;&lt;h2 id=&quot;agents-and-dashboards&quot;&gt;Agents and Dashboards&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;Shaper’s interactive dashboards are great for tracking known metrics or giving customers an overview of their data.&lt;/p&gt;
&lt;p&gt;But what do we do if we have &lt;strong&gt;ad-hoc questions&lt;/strong&gt; or want to &lt;strong&gt;explore our data&lt;/strong&gt; beyond what any existing dashboard was built for?&lt;/p&gt;
&lt;p&gt;That’s where AI agents can help:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Build an agent that answers your business team’s data questions without the help of a data analyst.&lt;/li&gt;
&lt;li&gt;Build an agent to explore new use cases for your data.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Shaper’s SQL-based approach means that instead of answering one-off questions your &lt;strong&gt;data team&lt;/strong&gt; can focus on:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Reviewing and verifying answers the agent gives.&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Turn any ad-hoc request into customer-facing dashboards and reusable reports&lt;/strong&gt;.&lt;/li&gt;
&lt;/ol&gt;
&lt;div&gt;&lt;h2 id=&quot;build-your-own-agent&quot;&gt;Build Your Own Agent&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;Building your own agent not only means that you stay in full control of your data, but also that you can build an agent that is truly useful in practice.
You can narrow the problem scope and build an agent for exactly the workflows most useful to you, integrating it into your processes with the context it needs.&lt;/p&gt;
&lt;p&gt;Shaper provides the foundation to access data, create charts and reports, while &lt;strong&gt;ensuring correctness and keeping everything reproducible&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;At Taleshape, we help our customers build systems exactly like this agent.&lt;/p&gt;
&lt;p&gt;If you’d like to learn more, don’t hesitate to &lt;a href=&quot;https://taleshape.com/contact&quot;&gt;reach out.&lt;/a&gt;&lt;/p&gt;
&lt;div&gt; &lt;h2&gt;Subscribe to Newsletter&lt;/h2&gt; &lt;p&gt;Get the new post delivered straight to your inbox.&lt;/p&gt; &lt;form id=&quot;blog-signup-form&quot; action=&quot;https://app.loops.so/api/newsletter-form/cmciwkod00u4a0iy59t1zcyk4&quot;&gt; &lt;div&gt; &lt;input id=&quot;blog-email&quot; type=&quot;email&quot; name=&quot;email&quot; required placeholder=&quot;Your email address&quot; autocomplete=&quot;email&quot;&gt; &lt;p&gt;&lt;/p&gt; &lt;/div&gt; &lt;button type=&quot;submit&quot;&gt;
Subscribe
&lt;/button&gt; &lt;/form&gt; &lt;p&gt;
Thank you for signing up!
&lt;/p&gt; &lt;p&gt;Unsubscribe at any time.&lt;/p&gt; &lt;/div&gt;  </content:encoded></item><item><title>Self-hosted Data Lake Platform with DuckDB, DuckLake and Shaper</title><link>https://taleshape.com/blog/simple-self-hosted-data-lake-platform-with-duckdb-ducklake-and-shaper/</link><guid isPermaLink="true">https://taleshape.com/blog/simple-self-hosted-data-lake-platform-with-duckdb-ducklake-and-shaper/</guid><pubDate>Tue, 31 Mar 2026 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;Do you need to &lt;strong&gt;store and analyze large amounts of data without breaking the bank&lt;/strong&gt;?
Do you need to run &lt;strong&gt;in your own infrastructure&lt;/strong&gt; instead of relying on the big cloud providers?&lt;/p&gt;
&lt;p&gt;You can build a simple data lake platform that scales &lt;strong&gt;from a $5 VM to petabytes of data&lt;/strong&gt; with these three open source tools: &lt;a href=&quot;https://duckdb.org/&quot;&gt;DuckDB&lt;/a&gt;, &lt;a href=&quot;https://ducklake.select/&quot;&gt;DuckLake&lt;/a&gt;, and &lt;a href=&quot;https://taleshape.com/shaper/docs&quot;&gt;Shaper&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://taleshape.com/_astro/ducklake-diagram.4jCthBfF_Z2qxIEd.webp&quot; alt=&quot;Architecture Diagram&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; fetchpriority=&quot;auto&quot; width=&quot;1408&quot; height=&quot;768&quot;&gt;&lt;/p&gt;
&lt;div&gt;&lt;h2 id=&quot;what-does-a-data-platform-need&quot;&gt;What Does a Data Platform Need?&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;A minimal data platform requires at least three components:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Data Storage&lt;/li&gt;
&lt;li&gt;Data Ingestion and Transformation&lt;/li&gt;
&lt;li&gt;Data Analysis and Visualization&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Let’s go over them one by one:&lt;/p&gt;
&lt;div&gt;&lt;h2 id=&quot;data-storage&quot;&gt;Data Storage&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;When you have only a small amount of data you can use DuckDB to directly query your production database such as &lt;strong&gt;Postgres&lt;/strong&gt;, but as your data grows that gets &lt;strong&gt;slow and expensive&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;One alternative is to replicate data to a &lt;strong&gt;data warehouse&lt;/strong&gt; such as Snowflake, BigQuery, Redshift or even Clickhouse. But warehouses are &lt;strong&gt;complex to operate and expensive&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;That’s where &lt;strong&gt;data lakes&lt;/strong&gt; come in:&lt;/p&gt;
&lt;div&gt;&lt;h3 id=&quot;whats-a-data-lake&quot;&gt;What’s a Data Lake?&lt;/h3&gt;&lt;/div&gt;
&lt;p&gt;Building a data lake architecture means &lt;strong&gt;storing data as files in a cheap object storage&lt;/strong&gt; such as S3.&lt;/p&gt;
&lt;p&gt;Data is then stored in a format optimized for analytics such as &lt;strong&gt;Parquet files&lt;/strong&gt;.
And DuckDB can query those files really efficiently - fast enough for most use cases even with large amounts of data.&lt;/p&gt;
&lt;p&gt;But &lt;strong&gt;Parquet files are read-only&lt;/strong&gt;. To update data you need to create a new file.
So you quickly end up with a large number of files that you need to manage and the need to rewrite files for better performance.&lt;/p&gt;
&lt;p&gt;That’s why a data lake needs a &lt;strong&gt;metadata layer&lt;/strong&gt; - a &lt;strong&gt;catalog&lt;/strong&gt; of all the data.&lt;/p&gt;
&lt;p&gt;Once you have a catalog, using a data lake feels just like using a database:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;You create, update and insert into tables without thinking about Parquet files.&lt;/strong&gt;
The catalog automatically rewrites and optimizes the files for you as needed.&lt;/p&gt;
&lt;div&gt;&lt;h3 id=&quot;why-ducklake&quot;&gt;Why DuckLake?&lt;/h3&gt;&lt;/div&gt;
&lt;p&gt;The most popular implementation of such a catalog is &lt;a href=&quot;https://iceberg.apache.org/&quot;&gt;Apache Iceberg&lt;/a&gt;.
Cloud providers offer Iceberg as a service (for example &lt;a href=&quot;https://duckdb.org/docs/current/core_extensions/iceberg/amazon_s3_tables&quot;&gt;AWS S3 Tables&lt;/a&gt; and &lt;a href=&quot;https://developers.cloudflare.com/r2/data-catalog/config-examples/duckdb/&quot;&gt;Cloudflare R2 Data Catalog&lt;/a&gt;).&lt;/p&gt;
&lt;p&gt;And DuckDB has great support for &lt;a href=&quot;https://duckdb.org/docs/current/core_extensions/iceberg/overview&quot;&gt;Iceberg&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;If your infrastructure provider already supports Iceberg on top of its object storage and you don’t mind paying the premium for that, that’s a great option for a data lake.&lt;/p&gt;
&lt;p&gt;However, to run your own data lake on top of any object storage, &lt;strong&gt;&lt;a href=&quot;https://ducklake.select/&quot;&gt;DuckLake&lt;/a&gt; is much simpler&lt;/strong&gt; than running your own Iceberg implementation.&lt;/p&gt;
&lt;p&gt;DuckLake doesn’t require any additional services to run. It works with any object storage and supports many databases for its catalog.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;If you have an S3-compatible service and a Postgres database, you can start using DuckLake without any new infrastructure.&lt;/strong&gt;&lt;/p&gt;
&lt;div&gt;&lt;h2 id=&quot;running-ducklake-and-shaper&quot;&gt;Running DuckLake and Shaper&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;We can run a complete data lake platform in a simple docker-compose setup:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Minio for object storage&lt;/li&gt;
&lt;li&gt;Postgres for the DuckLake catalog&lt;/li&gt;
&lt;li&gt;Shaper to run DuckDB and provide a UI for querying and scheduling jobs&lt;/li&gt;
&lt;/ul&gt;
&lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;span&gt;docker-compose.yml&lt;/span&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;services&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;minio&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;image&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;minio/minio&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;restart&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;unless-stopped&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;command&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;server /data --console-address &quot;:9001&quot;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;environment&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;MINIO_ROOT_USER&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;minioadmin&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;MINIO_ROOT_PASSWORD&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;minioadmin&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;volumes&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;      &lt;/span&gt;&lt;/span&gt;&lt;span&gt;- &lt;/span&gt;&lt;span&gt;minio_data:/data&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;healthcheck&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;test&lt;/span&gt;&lt;span&gt;: [&lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt;CMD&lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt;, &lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt;mc&lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt;, &lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt;ready&lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt;, &lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt;local&lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt;]&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;interval&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;5s&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;timeout&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;5s&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;retries&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;10&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;minio-init&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;image&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;minio/mc&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;depends_on&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;minio&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;        &lt;/span&gt;&lt;span&gt;condition&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;service_healthy&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;entrypoint&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;&gt;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;      &lt;/span&gt;&lt;/span&gt;&lt;span&gt;/bin/sh -c &quot;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;        &lt;/span&gt;&lt;/span&gt;&lt;span&gt;mc alias set local http://minio:9000 minioadmin minioadmin &amp;#x26;&amp;#x26;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;        &lt;/span&gt;&lt;/span&gt;&lt;span&gt;mc mb --ignore-existing local/ducklake &amp;#x26;&amp;#x26;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;        &lt;/span&gt;&lt;/span&gt;&lt;span&gt;echo &apos;Bucket ducklake ready.&apos;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;      &lt;/span&gt;&lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;restart&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt;no&lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;postgres&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;image&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;postgres:16&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;restart&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;unless-stopped&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;environment&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;POSTGRES_USER&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;ducklake&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;POSTGRES_PASSWORD&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;ducklake&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;POSTGRES_DB&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;ducklake&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;volumes&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;      &lt;/span&gt;&lt;/span&gt;&lt;span&gt;- &lt;/span&gt;&lt;span&gt;postgres_data:/var/lib/postgresql/data&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;healthcheck&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;test&lt;/span&gt;&lt;span&gt;: [&lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt;CMD-SHELL&lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt;, &lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt;pg_isready -U ducklake -d ducklake&lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt;]&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;interval&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;5s&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;timeout&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;5s&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;retries&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;10&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;shaper&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;image&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;taleshape/shaper&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;restart&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;unless-stopped&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;depends_on&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;postgres&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;        &lt;/span&gt;&lt;span&gt;condition&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;service_healthy&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;minio-init&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;        &lt;/span&gt;&lt;span&gt;condition&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;service_completed_successfully&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;ports&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;      &lt;/span&gt;&lt;/span&gt;&lt;span&gt;- &lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt;5454:5454&lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;# UI → http://localhost:5454&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;volumes&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;      &lt;/span&gt;&lt;/span&gt;&lt;span&gt;- &lt;/span&gt;&lt;span&gt;./init.sql:/init.sql:ro&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;environment&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;SHAPER_INIT_SQL_FILE&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;/init.sql&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;MINIO_KEY_ID&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;minioadmin&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;MINIO_SECRET&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;minioadmin&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;MINIO_ENDPOINT&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;minio:9000&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;MINIO_BUCKET&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;ducklake&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;PG_HOST&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;postgres&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;PG_PORT&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt;5432&lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;PG_DB&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;ducklake&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;PG_USER&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;ducklake&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;PG_PASSWORD&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;ducklake&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;volumes&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;minio_data&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;postgres_data&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;shaper_data&lt;/span&gt;&lt;span&gt;:&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;p&gt;We also need to configure Shaper to set up the secrets for DuckLake.
We can do that in a separate &lt;a href=&quot;https://taleshape.com/shaper/docs/connecting-to-data-sources/&quot;&gt;&lt;code dir=&quot;auto&quot;&gt;init.sql&lt;/code&gt;&lt;/a&gt; file that Shaper executes on startup:&lt;/p&gt;
&lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;span&gt;init.sql&lt;/span&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;CREATE&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;OR&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;REPLACE&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;SECRET&lt;/span&gt;&lt;span&gt; minio_secret (&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;TYPE&lt;/span&gt;&lt;span&gt; s3,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;KEY_ID &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;${MINIO_KEY_ID}&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;SECRET&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;${MINIO_SECRET}&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;REGION &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;us-east-1&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;ENDPOINT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;${MINIO_ENDPOINT}&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;URL_STYLE &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;path&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;USE_SSL false&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;);&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;CREATE&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;OR&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;REPLACE&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;SECRET&lt;/span&gt;&lt;span&gt; pg_secret (&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;TYPE&lt;/span&gt;&lt;span&gt; postgres,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;HOST &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;${PG_HOST}&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;PORT ${PG_PORT},&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;DATABASE&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;${PG_DB}&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;USER &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;${PG_USER}&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;PASSWORD&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;${PG_PASSWORD}&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;);&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;CREATE&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;SECRET&lt;/span&gt;&lt;span&gt; ducklake_secret (&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;TYPE&lt;/span&gt;&lt;span&gt; ducklake,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;METADATA_PATH &lt;/span&gt;&lt;span&gt;&apos;&apos;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;DATA_PATH &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;s3://${MINIO_BUCKET}/&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;METADATA_PARAMETERS MAP {&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;TYPE&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;postgres&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;, &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;SECRET&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;pg_secret&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;}&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;);&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;ATTACH&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;ducklake:ducklake_secret&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; ducklake;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;p&gt;You can start the setup by running &lt;code dir=&quot;auto&quot;&gt;docker compose up -d&lt;/code&gt; in the directory where you created the &lt;code dir=&quot;auto&quot;&gt;docker-compose.yml&lt;/code&gt; and &lt;code dir=&quot;auto&quot;&gt;init.sql&lt;/code&gt; files.&lt;/p&gt;
&lt;div&gt;&lt;h2 id=&quot;ingesting-and-transforming-data&quot;&gt;Ingesting and Transforming Data&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;We can use Shaper’s &lt;a href=&quot;https://taleshape.com/shaper/docs/tasks-and-scheduling/&quot;&gt;Task feature&lt;/a&gt; as a simple scheduling system to run SQL queries that ingest and transform data in our data lake:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Open &lt;a href=&quot;http://localhost:5454&quot;&gt;http://localhost:5454&lt;/a&gt; in your browser.&lt;/li&gt;
&lt;li&gt;Click on &lt;strong&gt;New&lt;/strong&gt; and select &lt;strong&gt;Task&lt;/strong&gt; at the top.&lt;/li&gt;
&lt;li&gt;Copy and paste the following SQL:&lt;/li&gt;
&lt;/ol&gt;
&lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;NULL&lt;/span&gt;&lt;span&gt;::SCHEDULE;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;CREATE&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;TABLE&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;IF&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;NOT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;EXISTS&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;ducklake&lt;/span&gt;&lt;span&gt;.&lt;/span&gt;&lt;span&gt;sessions&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;*&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;FROM&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;https://taleshape.com/sample-data/sessions.csv&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;ol start=&quot;4&quot;&gt;
&lt;li&gt;Click on “Create” at the top right and give the task a name such as “Ingest Sessions Data”.&lt;/li&gt;
&lt;li&gt;Now click on “Run” in the top right corner.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Congrats, you’ve loaded your first data into DuckLake!&lt;/p&gt;
&lt;p&gt;For real use cases you would want to schedule tasks to run automatically and then you would use for example &lt;a href=&quot;https://duckdb.org/docs/current/sql/statements/merge_into&quot;&gt;&lt;code dir=&quot;auto&quot;&gt;MERGE INTO&lt;/code&gt;&lt;/a&gt; to update your data lake continuously.&lt;/p&gt;
&lt;p&gt;And of course not only Shaper can access DuckLake. For more complex setups you can introduce other tools for your data pipelines such as &lt;a href=&quot;https://www.getdbt.com/&quot;&gt;DBT&lt;/a&gt; and &lt;a href=&quot;https://dagster.io/&quot;&gt;Dagster&lt;/a&gt;.&lt;/p&gt;
&lt;div&gt;&lt;h2 id=&quot;analyze-and-visualize-data&quot;&gt;Analyze and Visualize Data&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;Once we have data in our data lake we can now query it.&lt;/p&gt;
&lt;p&gt;Let’s build a first Shaper dashboard on top of DuckLake:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;In the Shaper UI click on &lt;strong&gt;New&lt;/strong&gt; and select &lt;strong&gt;Dashboard&lt;/strong&gt; at the top.&lt;/li&gt;
&lt;li&gt;Copy and paste the following SQL:&lt;/li&gt;
&lt;/ol&gt;
&lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;count&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;*&lt;/span&gt;&lt;span&gt;) &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt;Total Sessions&lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;FROM&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;ducklake&lt;/span&gt;&lt;span&gt;.&lt;/span&gt;&lt;span&gt;sessions&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;Sessions per Week&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;::LABEL;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;  &lt;/span&gt;&lt;/span&gt;&lt;span&gt;date_trunc(&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;week&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;, created_at)::XAXIS,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;  &lt;/span&gt;&lt;/span&gt;&lt;span&gt;category::CATEGORY,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;count&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;*&lt;/span&gt;&lt;span&gt;)::BARCHART_STACKED,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;FROM&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;ducklake&lt;/span&gt;&lt;span&gt;.&lt;/span&gt;&lt;span&gt;sessions&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;GROUP BY&lt;/span&gt;&lt;span&gt; ALL &lt;/span&gt;&lt;span&gt;ORDER BY&lt;/span&gt;&lt;span&gt; ALL;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;ol start=&quot;3&quot;&gt;
&lt;li&gt;Click on “Create” at the top right and give the dashboard a name such as “Sessions Dashboard”.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Congrats, you just built your first dashboard on top of your data lake!&lt;/p&gt;
&lt;p&gt;You can learn more about building dashboards in Shaper in the &lt;a href=&quot;https://taleshape.com/shaper/docs/getting-started/&quot;&gt;Getting Started Guide&lt;/a&gt;.&lt;/p&gt;
&lt;div&gt;&lt;h2 id=&quot;deploying-to-production&quot;&gt;Deploying to Production&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;You now have a minimal, but complete data lake platform that you can build interesting data products on top of.&lt;/p&gt;
&lt;p&gt;But for now this is only running on your computer locally. Now it’s time to deploy this setup into a production infrastructure.&lt;/p&gt;
&lt;p&gt;The cheapest way to get started is to simply run all services on a single server, almost identical to how we did above.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;But you wouldn’t need a data lake if you have so little data that it fits on a single machine&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;Instead, you want to make use of an &lt;strong&gt;existing object storage service&lt;/strong&gt; which almost every hosting provider offers.&lt;/p&gt;
&lt;div&gt;&lt;h3 id=&quot;recommended-hetzner&quot;&gt;Recommended: Hetzner&lt;/h3&gt;&lt;/div&gt;
&lt;p&gt;If you are looking for a &lt;strong&gt;more affordable&lt;/strong&gt; hosting provider or prefer a &lt;strong&gt;European&lt;/strong&gt; provider over the big American cloud providers, I recommend you look into Hetzner (&lt;a href=&quot;https://hetzner.cloud/?ref=mbYP5pBufIAn&quot;&gt;referral&lt;/a&gt; to get &lt;strong&gt;free 20€&lt;/strong&gt;).&lt;/p&gt;
&lt;p&gt;You get a VM for &lt;strong&gt;3.49€&lt;/strong&gt; and 1TB of object storage for &lt;strong&gt;4.99€&lt;/strong&gt;, hosted in Germany.&lt;/p&gt;
&lt;p&gt;Just be careful that there are some &lt;strong&gt;gotchas with Hetzner’s object storage&lt;/strong&gt;. To make sure your setup is properly secured, check out this &lt;strong&gt;detailed guide by my friend Floyd&lt;/strong&gt;:&lt;/p&gt;
&lt;div&gt; &lt;span&gt; &lt;a href=&quot;https://berndsen.io/blog/0402-ducklake-hetzner/&quot;&gt; &lt;span&gt;DuckLake on Hetzner&lt;/span&gt; &lt;/a&gt;  &lt;/span&gt; &lt;svg aria-hidden=&quot;true&quot; width=&quot;16&quot; height=&quot;16&quot; viewBox=&quot;0 0 24 24&quot; fill=&quot;currentColor&quot;&gt;&lt;path d=&quot;M17.92 11.62a1.001 1.001 0 0 0-.21-.33l-5-5a1.003 1.003 0 1 0-1.42 1.42l3.3 3.29H7a1 1 0 0 0 0 2h7.59l-3.3 3.29a1.002 1.002 0 0 0 .325 1.639 1 1 0 0 0 1.095-.219l5-5a1 1 0 0 0 .21-.33 1 1 0 0 0 0-.76Z&quot;&gt;&lt;/path&gt;&lt;/svg&gt; &lt;/div&gt; 
&lt;hr&gt;
&lt;p&gt;And if you are looking for &lt;strong&gt;fully-managed Shaper connected to your data lake and integrated into your systems&lt;/strong&gt;, check out &lt;a href=&quot;https://taleshape.com/plans-and-pricing/&quot;&gt;Taleshape’s Managed Hosting and Support Plans&lt;/a&gt;.&lt;/p&gt;
&lt;div&gt; &lt;h2&gt;Subscribe to Newsletter&lt;/h2&gt; &lt;p&gt;Get the new post delivered straight to your inbox.&lt;/p&gt; &lt;form id=&quot;blog-signup-form&quot; action=&quot;https://app.loops.so/api/newsletter-form/cmciwkod00u4a0iy59t1zcyk4&quot;&gt; &lt;div&gt; &lt;input id=&quot;blog-email&quot; type=&quot;email&quot; name=&quot;email&quot; required placeholder=&quot;Your email address&quot; autocomplete=&quot;email&quot;&gt; &lt;p&gt;&lt;/p&gt; &lt;/div&gt; &lt;button type=&quot;submit&quot;&gt;
Subscribe
&lt;/button&gt; &lt;/form&gt; &lt;p&gt;
Thank you for signing up!
&lt;/p&gt; &lt;p&gt;Unsubscribe at any time.&lt;/p&gt; &lt;/div&gt;  </content:encoded></item><item><title>Analytics Dashboards as Code</title><link>https://taleshape.com/blog/analytics-dashboards-as-code/</link><guid isPermaLink="true">https://taleshape.com/blog/analytics-dashboards-as-code/</guid><pubDate>Tue, 09 Dec 2025 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;a href=&quot;https://github.com/taleshape-com/shaper&quot;&gt;Shaper&lt;/a&gt; has always been about &lt;strong&gt;building dashboards using SQL&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;And it’s just natural to manage &lt;strong&gt;SQL as files&lt;/strong&gt; just like any other code.&lt;/p&gt;
&lt;p&gt;With the latest release of Shaper you can now &lt;strong&gt;deploy dashboards from files&lt;/strong&gt;,&lt;br&gt;and &lt;strong&gt;live-preview dashboard changes&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;You get to:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;use your favorite editor - including its AI features&lt;/li&gt;
&lt;li&gt;track dashboards in Git&lt;/li&gt;
&lt;li&gt;collaborate with your team using pull requests&lt;/li&gt;
&lt;li&gt;deploy dashboards via CI/CD&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The workflow in action:&lt;/p&gt;
&lt;lite-youtube videoid=&quot;HHYvx_MsQHc&quot;&gt; &lt;a href=&quot;https://youtube.com/watch?v=HHYvx_MsQHc&quot;&gt; &lt;span&gt;Play&lt;/span&gt; &lt;/a&gt; &lt;/lite-youtube&gt;  
&lt;div&gt;&lt;h2 id=&quot;getting-started&quot;&gt;Getting Started&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;Getting started and sharing Shaper projects is now simpler than ever since everything is just files.
Let’s give it a try:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Clone &lt;a href=&quot;https://github.com/taleshape-com/demo-project/&quot;&gt;this&lt;/a&gt; demo Git repository:
&lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;span&gt;&lt;/span&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;git@github.com:taleshape-com/demo-project.git&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;/li&gt;
&lt;li&gt;Install Shaper in the project directory:
&lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;span&gt;&lt;/span&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;make&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;install&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;/li&gt;
&lt;li&gt;Start a local Shaper server to serve the dashboards:
&lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;span&gt;&lt;/span&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;make&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;serve&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;/li&gt;
&lt;li&gt;Deploy the two dashboards in the &lt;code dir=&quot;auto&quot;&gt;dashboards/&lt;/code&gt; folder by running the deploy command in a second terminal:
&lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;span&gt;&lt;/span&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;make&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;deploy&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;/li&gt;
&lt;li&gt;Live-preview changes by running the dev file watcher in the second terminal:
&lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;span&gt;&lt;/span&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;make&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;dev&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;br&gt;Now edit or create any SQL file in the &lt;code dir=&quot;auto&quot;&gt;dashboards/&lt;/code&gt; folder and see the changes live in your browser.&lt;/li&gt;
&lt;/ol&gt;
&lt;div&gt;&lt;h2 id=&quot;learn-more&quot;&gt;Learn More&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;Usually you will run Shaper on a server. Then you only need &lt;code dir=&quot;auto&quot;&gt;shaper dev&lt;/code&gt; locally when developing dashboards.
Once you configured authentication for your Shaper instance, the &lt;code dir=&quot;auto&quot;&gt;dev&lt;/code&gt; command will automatically prompt you to login and authenticate.
And instead of running &lt;code dir=&quot;auto&quot;&gt;deploy&lt;/code&gt; manually, you can use the &lt;a href=&quot;https://github.com/taleshape-com/shaper-deploy-action&quot;&gt;Shaper Github Action&lt;/a&gt; to deploy dashboards automatically in your CI/CD pipeline.&lt;/p&gt;
&lt;p&gt;All details are in the &lt;a href=&quot;https://taleshape.com/shaper/docs/file-based-workflow/&quot;&gt;documentation&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;And as always - we are happy to hear about any feedback via socials or Github &lt;a href=&quot;https://github.com/taleshape-com/shaper/issues&quot;&gt;issues&lt;/a&gt; or &lt;a href=&quot;https://github.com/taleshape-com/shaper/discussions&quot;&gt;discussions&lt;/a&gt;!&lt;/p&gt;
&lt;div&gt; &lt;h2&gt;Subscribe to Newsletter&lt;/h2&gt; &lt;p&gt;Get the new post delivered straight to your inbox.&lt;/p&gt; &lt;form id=&quot;blog-signup-form&quot; action=&quot;https://app.loops.so/api/newsletter-form/cmciwkod00u4a0iy59t1zcyk4&quot;&gt; &lt;div&gt; &lt;input id=&quot;blog-email&quot; type=&quot;email&quot; name=&quot;email&quot; required placeholder=&quot;Your email address&quot; autocomplete=&quot;email&quot;&gt; &lt;p&gt;&lt;/p&gt; &lt;/div&gt; &lt;button type=&quot;submit&quot;&gt;
Subscribe
&lt;/button&gt; &lt;/form&gt; &lt;p&gt;
Thank you for signing up!
&lt;/p&gt; &lt;p&gt;Unsubscribe at any time.&lt;/p&gt; &lt;/div&gt;  </content:encoded></item><item><title>Getting Started Building a Data Platform</title><link>https://taleshape.com/blog/getting-started-building-a-data-platform/</link><guid isPermaLink="true">https://taleshape.com/blog/getting-started-building-a-data-platform/</guid><pubDate>Thu, 16 Oct 2025 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;Ever wonder what a data platform is and if your company needs one?&lt;/p&gt;
&lt;p&gt;If the idea of hiring a data team to build and manage an enterprise data platform feels overwhelming, you’re not alone.&lt;/p&gt;
&lt;p&gt;Let me break down how you can get started from zero and &lt;strong&gt;build up data capabilities&lt;/strong&gt; at your company, one step at a time.&lt;/p&gt;
&lt;div&gt;&lt;h2 id=&quot;who-needs-a-data-platform&quot;&gt;Who Needs a Data Platform?&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;Nowadays, every company is a data company. From marketing and sales to product usage and customer support, all aspects of your business generate data.&lt;/p&gt;
&lt;p&gt;And that data is waiting to be activated. Turn it into reports to drive decisions. Build dashboards to support operations. Offer new product features and services to your customers that are directly driven by data and automation.&lt;/p&gt;
&lt;p&gt;Getting started is much more &lt;strong&gt;a cultural challenge&lt;/strong&gt; than a technical one.&lt;/p&gt;
&lt;p&gt;Start small. Make sure you see &lt;strong&gt;first successes by doing the work manually&lt;/strong&gt; without worrying about big investments in making it scalable.&lt;/p&gt;
&lt;p&gt;But once you see concrete value and it becomes painfully clear that technology is holding you back, you know it’s time to build out your data capabilities.&lt;/p&gt;
&lt;p&gt;Where do you go from here? Can you buy an off-the-shelf solution? Do you hire a data engineer? Do you need a dedicated data team or can your existing engineers handle it?&lt;/p&gt;
&lt;div&gt;&lt;h2 id=&quot;breaking-it-down&quot;&gt;Breaking It Down&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;You can break down data infrastructure into four main layers:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Ingestion:&lt;/strong&gt; Connect data sources, extract data and load it into a central repository&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Storage:&lt;/strong&gt; Store data in a structured format that is optimized for analytical workloads&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Transformation:&lt;/strong&gt; Clean, enrich, and transform data to make it practical to work with and ensure consistent definitions of key metrics across the organization&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Analytics:&lt;/strong&gt; Create dashboards, reports, and alerts to share insights internally and with your customers and partners&lt;/li&gt;
&lt;/ol&gt;
&lt;div&gt;&lt;h2 id=&quot;one-step-at-a-time&quot;&gt;One Step at a Time&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;There are many different tools to address all of these layers.&lt;/p&gt;
&lt;p&gt;Focus on solving &lt;strong&gt;concrete problems&lt;/strong&gt; you are experiencing and add new tools only when they directly solve a problem.&lt;/p&gt;
&lt;div&gt;&lt;h3 id=&quot;ingestion&quot;&gt;Ingestion&lt;/h3&gt;&lt;/div&gt;
&lt;p&gt;Start with &lt;strong&gt;querying your data directly where it is&lt;/strong&gt;. Introduce tools to load data into a central repository only when the complexity and volume make this impractical.&lt;/p&gt;
&lt;p&gt;You don’t need to address all data sources at once. Focus on the ones creating problems. &lt;strong&gt;Accept manual workarounds&lt;/strong&gt; when practical.&lt;/p&gt;
&lt;p&gt;Your data tooling should be able to query data across different data sources. You don’t need to worry about ingestion if directly querying a Postgres database and a Google Sheet gets the job done.&lt;/p&gt;
&lt;div&gt;&lt;h3 id=&quot;storage&quot;&gt;Storage&lt;/h3&gt;&lt;/div&gt;
&lt;p&gt;Chances are you already store your data in a database such as PostgreSQL or MySQL. If you’re not having performance issues, there’s no need to introduce a separate database for analytical workloads.&lt;/p&gt;
&lt;p&gt;Only if &lt;strong&gt;performance or cost becomes an issue&lt;/strong&gt; should you start addressing it.&lt;/p&gt;
&lt;p&gt;Storage is a critical component since it’s where the actual data lives. &lt;strong&gt;Data outlives applications&lt;/strong&gt; built on top of it. Pick an established and open standard to store data.&lt;/p&gt;
&lt;p&gt;Keep in mind that there is no one-size-fits-all solution. You might need multiple data stores &lt;strong&gt;optimized for different use cases&lt;/strong&gt;. You’ll know what you’re looking for when you act on concrete problems instead of trying to find a solution for hypothetical future problems.&lt;/p&gt;
&lt;div&gt;&lt;h3 id=&quot;transformation&quot;&gt;Transformation&lt;/h3&gt;&lt;/div&gt;
&lt;p&gt;Start delivering value before adding a separate data transformation step. Introduce a dedicated data transformation layer when &lt;strong&gt;queries start taking too long&lt;/strong&gt;, or &lt;strong&gt;metrics become unreliable&lt;/strong&gt; and hard to maintain because the &lt;strong&gt;same logic is repeated in many places&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;A few &lt;strong&gt;materialized views&lt;/strong&gt; in your database can take you a long way.&lt;/p&gt;
&lt;p&gt;You’ll know it’s time to look into real-time stream processing, data lineage and orchestration tools once you experience the issues that these tools are designed to solve.&lt;/p&gt;
&lt;div&gt;&lt;h3 id=&quot;analytics&quot;&gt;Analytics&lt;/h3&gt;&lt;/div&gt;
&lt;p&gt;Many software companies start out by building custom analytics features. As you use data to drive operations and user-facing functionality, building custom solutions for every new workflow and view on the data becomes slow and expensive.&lt;/p&gt;
&lt;p&gt;Introduce a data visualization tool to &lt;strong&gt;quickly build analytics dashboards&lt;/strong&gt; and reports. This is a &lt;strong&gt;great first step&lt;/strong&gt; and enables a single &lt;strong&gt;data analyst&lt;/strong&gt; to deliver a lot of value without introducing any other data infrastructure.&lt;/p&gt;
&lt;div&gt;&lt;h2 id=&quot;getting-started&quot;&gt;Getting Started&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;I built &lt;a href=&quot;https://taleshape.com/shaper/docs&quot;&gt;Shaper&lt;/a&gt; to help companies in exactly this situation.&lt;/p&gt;
&lt;div&gt;&lt;a href=&quot;https://github.com/taleshape-com/shaper&quot; target=&quot;_blank&quot;&gt;  taleshape-com/shaper &lt;svg aria-hidden=&quot;true&quot; width=&quot;16&quot; height=&quot;16&quot; viewBox=&quot;0 0 24 24&quot; fill=&quot;currentColor&quot;&gt;&lt;path d=&quot;M12 .3a12 12 0 0 0-3.8 23.38c.6.12.83-.26.83-.57L9 21.07c-3.34.72-4.04-1.61-4.04-1.61-.55-1.39-1.34-1.76-1.34-1.76-1.08-.74.09-.73.09-.73 1.2.09 1.83 1.24 1.83 1.24 1.08 1.83 2.81 1.3 3.5 1 .1-.78.42-1.31.76-1.61-2.67-.3-5.47-1.33-5.47-5.93 0-1.31.47-2.38 1.24-3.22-.14-.3-.54-1.52.1-3.18 0 0 1-.32 3.3 1.23a11.5 11.5 0 0 1 6 0c2.28-1.55 3.29-1.23 3.29-1.23.64 1.66.24 2.88.12 3.18a4.65 4.65 0 0 1 1.23 3.22c0 4.61-2.8 5.63-5.48 5.92.42.36.81 1.1.81 2.22l-.01 3.29c0 .31.2.69.82.57A12 12 0 0 0 12 .3Z&quot;&gt;&lt;/path&gt;&lt;/svg&gt; &lt;/a&gt; &lt;/div&gt;
&lt;p&gt;Shaper is a simple interface on top of &lt;a href=&quot;https://duckdb.org/&quot;&gt;DuckDB&lt;/a&gt; that allows you to build analytics dashboards and automate data workflows using &lt;strong&gt;only SQL&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;Thanks to DuckDB, it’s easy to query data across various sources ranging from databases to CSV files and Google Sheets.&lt;/p&gt;
&lt;p&gt;You can go a long way before having to add more layers to your data stack.&lt;/p&gt;
&lt;p&gt;Give it a try and let me know what you think.&lt;/p&gt;
&lt;div&gt; &lt;h2&gt;Subscribe to Newsletter&lt;/h2&gt; &lt;p&gt;Get the new post delivered straight to your inbox.&lt;/p&gt; &lt;form id=&quot;blog-signup-form&quot; action=&quot;https://app.loops.so/api/newsletter-form/cmciwkod00u4a0iy59t1zcyk4&quot;&gt; &lt;div&gt; &lt;input id=&quot;blog-email&quot; type=&quot;email&quot; name=&quot;email&quot; required placeholder=&quot;Your email address&quot; autocomplete=&quot;email&quot;&gt; &lt;p&gt;&lt;/p&gt; &lt;/div&gt; &lt;button type=&quot;submit&quot;&gt;
Subscribe
&lt;/button&gt; &lt;/form&gt; &lt;p&gt;
Thank you for signing up!
&lt;/p&gt; &lt;p&gt;Unsubscribe at any time.&lt;/p&gt; &lt;/div&gt;  </content:encoded></item><item><title>Build Your Own Bluesky Analytics Dashboard</title><link>https://taleshape.com/blog/build-your-own-bluesky-analytics-dashboard/</link><guid isPermaLink="true">https://taleshape.com/blog/build-your-own-bluesky-analytics-dashboard/</guid><pubDate>Thu, 28 Aug 2025 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Are you using Bluesky and want to stay on top of what’s happening?
Are you curious how you can use Shaper to pull data from APIs and build interactive dashboards, all in a single tool and using just SQL?&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Let’s automate pulling posts data from the Bluesky API to track topics we are interested in,
and then create a data dashboard that visualizes activity around these topics.&lt;/p&gt;
&lt;p&gt;You will get a dashboard that looks like this:&lt;/p&gt;
&lt;a href=&quot;https://demo.taleshape.com/view/mqn0sj9gqaxcz0kp7ulbfoel&quot; target=&quot;_blank&quot;&gt;&lt;img src=&quot;https://taleshape.com/_astro/bluesky_demo_dashboard.CkimwIlB_Z1VQ8Bk.webp&quot; alt=&quot;Hero Image&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; fetchpriority=&quot;auto&quot; width=&quot;1920&quot; height=&quot;957&quot;&gt;&lt;div&gt;&lt;button&gt;Click to see it live&lt;/button&gt;&lt;/div&gt;&lt;svg aria-hidden=&quot;true&quot; width=&quot;16&quot; height=&quot;16&quot; viewBox=&quot;0 0 24 24&quot; fill=&quot;currentColor&quot;&gt;&lt;path d=&quot;M19.33 10.18a1 1 0 0 1-.77 0 1 1 0 0 1-.62-.93l.01-1.83-8.2 8.2a1 1 0 0 1-1.41-1.42l8.2-8.2H14.7a1 1 0 0 1 0-2h4.25a1 1 0 0 1 1 1v4.25a1 1 0 0 1-.62.93Z&quot;&gt;&lt;/path&gt;&lt;path d=&quot;M11 4a1 1 0 1 1 0 2H7a1 1 0 0 0-1 1v10a1 1 0 0 0 1 1h10a1 1 0 0 0 1-1v-4a1 1 0 1 1 2 0v4a3 3 0 0 1-3 3H7a3 3 0 0 1-3-3V7a3 3 0 0 1 3-3h4Z&quot;&gt;&lt;/path&gt;&lt;/svg&gt;&lt;/a&gt;
&lt;div&gt;&lt;h2 id=&quot;lets-get-started&quot;&gt;Let’s Get Started&lt;/h2&gt;&lt;/div&gt;
&lt;ol role=&quot;list&quot;&gt;
&lt;li&gt;Let’s open a terminal, create a new directory, and change into it:
&lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;span&gt;&lt;/span&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;mkdir&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;bluesky-dashboard&lt;/span&gt;&lt;span&gt; &amp;#x26;&amp;#x26; &lt;/span&gt;&lt;span&gt;cd&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;bluesky-dashboard&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;/li&gt;
&lt;li&gt;You will need credentials to authenticate with the Bluesky API.&lt;br&gt;
Create a &lt;a href=&quot;https://bsky.app/settings/app-passwords&quot;&gt;Bluesky App Password&lt;/a&gt; and save it together with your handle as &lt;code dir=&quot;auto&quot;&gt;bluesky_credentials.json&lt;/code&gt;:
&lt;br&gt;
&lt;br&gt;
&lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;span&gt;&lt;/span&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;echo&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;{ &quot;identifier&quot;: &quot;&quot;, &quot;password&quot;: &quot;&quot; }&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&gt;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;bluesky_credentials.json&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;/li&gt;
&lt;li&gt;Now let’s run Shaper. The easiest way is to run it via Docker or NPM:
&lt;starlight-tabs&gt; &lt;div&gt; &lt;ul role=&quot;tablist&quot;&gt; &lt;li role=&quot;presentation&quot;&gt; &lt;a role=&quot;tab&quot; href=&quot;#tab-panel-6&quot; id=&quot;tab-6&quot; aria-selected=&quot;true&quot; tabindex=&quot;0&quot;&gt; &lt;svg aria-hidden=&quot;true&quot; width=&quot;16&quot; height=&quot;16&quot; viewBox=&quot;0 0 24 24&quot; fill=&quot;currentColor&quot;&gt;&lt;path d=&quot;M10.109 6.473L10.109 4.427L12.188 4.427L12.188 6.473L10.109 6.473ZM10.109 8.849L10.109 6.803L12.188 6.803L12.188 8.849L10.109 8.849ZM10.109 11.324L10.109 9.277L12.188 9.277L12.188 11.324L10.109 11.324ZM7.733 8.849L7.733 6.803L9.812 6.803L9.812 8.849L7.733 8.849ZM7.733 11.324L7.733 9.277L9.812 9.277L9.812 11.324L7.733 11.324ZM5.258 8.849L5.258 6.803L7.337 6.803L7.337 8.849L5.258 8.849ZM5.258 11.324L5.258 9.277L7.337 9.277L7.337 11.324L5.258 11.324ZM2.882 11.324L2.882 9.277L4.961 9.277L4.961 11.324L2.882 11.324ZM12.584 11.324L12.584 9.277L14.663 9.277L14.663 11.324L12.584 11.324ZM23.111 9.575L23.111 9.575Q22.286 9.377 21.758 9.377L21.758 9.377Q21.032 9.509 20.735 8.750Q20.438 7.991 19.811 7.495L19.811 7.495Q19.349 6.968 18.920 7.083Q18.491 7.198 18.260 7.826L18.260 7.826Q17.963 8.816 18.161 10.103L18.161 10.103Q18.326 10.796 18.177 11.043Q18.029 11.291 17.319 11.456Q16.610 11.620 8.492 11.654L8.492 11.654Q4.433 11.654 0.539 11.620L0.539 11.620L0.539 11.620Q0.374 11.620 0.291 11.934Q0.209 12.248 0.209 12.677L0.209 12.677Q0.209 13.238 0.308 13.997L0.308 13.997Q0.407 14.987 0.638 15.449L0.638 15.449Q1.430 17.230 2.717 18.287L2.717 18.287Q4.136 19.442 5.984 19.574L5.984 19.574L10.637 19.574Q12.782 19.508 14.696 18.485L14.696 18.485Q16.412 17.593 18.062 15.878L18.062 15.878Q19.382 14.360 20.108 12.776L20.108 12.776Q20.405 12.215 21.065 12.082Q21.725 11.951 22.055 11.852L22.055 11.852Q22.550 11.720 22.913 11.422L22.913 11.422Q23.012 11.225 23.309 10.928L23.309 10.928Q23.837 10.532 23.787 10.136Q23.738 9.739 23.111 9.575ZM7.733 14.822L7.733 14.822Q8.063 14.822 8.310 15.053Q8.558 15.284 8.558 15.663Q8.558 16.043 8.343 16.257Q8.129 16.471 7.766 16.471Q7.403 16.471 7.155 16.257Q6.908 16.043 6.908 15.663Q6.908 15.284 7.155 15.053Q7.403 14.822 7.733 14.822ZM2.684 17.099L2.684 17.099Q3.047 17.066 3.740 17.099L3.740 17.099Q4.664 17.132 5.060 17.000L5.060 17.000Q5.819 16.736 6.330 16.917Q6.842 17.099 7.238 17.825L7.238 17.825Q7.403 18.188 7.799 18.485L7.799 18.485Q8.030 18.650 8.624 18.980L8.624 18.980L8.987 19.178Q7.271 19.409 5.555 18.831Q3.839 18.254 2.684 17.099Z&quot;&gt;&lt;/path&gt;&lt;/svg&gt; Docker &lt;/a&gt; &lt;/li&gt;&lt;li role=&quot;presentation&quot;&gt; &lt;a role=&quot;tab&quot; href=&quot;#tab-panel-7&quot; id=&quot;tab-7&quot; aria-selected=&quot;false&quot; tabindex=&quot;-1&quot;&gt; &lt;svg aria-hidden=&quot;true&quot; width=&quot;16&quot; height=&quot;16&quot; viewBox=&quot;0 0 24 24&quot; fill=&quot;currentColor&quot;&gt;&lt;path d=&quot;M24 7.296L0 7.296L0 15.296L6.816 15.296L6.816 16.704L12.096 16.704L12.096 15.392L24 15.392L24 7.296ZM6.592 8.704L6.592 13.984L5.312 13.984L5.312 10.112L4 10.112L4 13.984L1.312 13.984L1.312 8.704L6.592 8.704ZM13.184 13.984L13.216 13.984L10.496 13.984L10.496 15.392L7.808 15.392L7.808 8.800L13.088 8.800Q13.216 10.400 13.184 13.984L13.184 13.984ZM22.592 8.704L22.592 13.984L21.312 13.984L21.312 10.112L20 10.112L20 13.984L18.592 13.984L18.592 10.112L17.312 10.112L17.312 13.984L14.592 13.984L14.592 8.704L22.592 8.704ZM11.904 12.704L11.904 10.112L10.592 10.112L10.592 12.704L11.904 12.704Z&quot;&gt;&lt;/path&gt;&lt;/svg&gt; npm &lt;/a&gt; &lt;/li&gt; &lt;/ul&gt; &lt;/div&gt; &lt;div id=&quot;tab-panel-6&quot; aria-labelledby=&quot;tab-6&quot; role=&quot;tabpanel&quot;&gt; &lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;span&gt;&lt;/span&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;docker&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;run&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;--rm&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;-it&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;-p5454:5454&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;\&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;-v&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;./bluesky_credentials.json:/bluesky_credentials.json&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;\&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;-v&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;./data:/data&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;taleshape/shaper&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;\&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;--init-sql&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt;INSTALL http_client FROM community; LOAD http_client;&lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt; &lt;/div&gt;&lt;div id=&quot;tab-panel-7&quot; aria-labelledby=&quot;tab-7&quot; role=&quot;tabpanel&quot; hidden&gt; &lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;span&gt;&lt;/span&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;npx&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;@taleshape/shaper&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;-d&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;./data&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt; &lt;/div&gt;  &lt;/starlight-tabs&gt;  
&lt;/li&gt;
&lt;li&gt;Open &lt;a href=&quot;http://localhost:5454&quot;&gt;http://localhost:5454&lt;/a&gt; in your browser and click on &lt;strong&gt;New&lt;/strong&gt;.&lt;br&gt;
Now let’s create a Task to fetch posts from Bluesky and store them in a database table.
Select &lt;strong&gt;Task&lt;/strong&gt; in the dropdown at the top of the page and paste in the following SQL code:
&lt;br&gt;
&lt;br&gt;
&lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;span&gt; (date_trunc(&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;hour&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;, &lt;/span&gt;&lt;span&gt;now&lt;/span&gt;&lt;span&gt;()) &lt;/span&gt;&lt;span&gt;+&lt;/span&gt;&lt;span&gt; INTERVAL &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;1h&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;)::SCHEDULE;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;CREATE&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;SCHEMA&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;IF&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;NOT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;EXISTS&lt;/span&gt;&lt;span&gt; bsky;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;CREATE&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;TABLE&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;IF&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;NOT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;EXISTS&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;bsky&lt;/span&gt;&lt;span&gt;.&lt;/span&gt;&lt;span&gt;posts&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;topic &lt;/span&gt;&lt;span&gt;VARCHAR&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;created_at &lt;/span&gt;&lt;span&gt;TIMESTAMP&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;cid &lt;/span&gt;&lt;span&gt;VARCHAR&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;author_handle &lt;/span&gt;&lt;span&gt;VARCHAR&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;url&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;VARCHAR&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;text&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;VARCHAR&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;like_count &lt;/span&gt;&lt;span&gt;INT&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;reply_count &lt;/span&gt;&lt;span&gt;INT&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;quote_count &lt;/span&gt;&lt;span&gt;INT&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;repost_count &lt;/span&gt;&lt;span&gt;INT&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;loaded_at &lt;/span&gt;&lt;span&gt;TIMESTAMP&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;DEFAULT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;now&lt;/span&gt;&lt;span&gt;(),&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;);&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SET&lt;/span&gt;&lt;span&gt; VARIABLE access_jwt &lt;/span&gt;&lt;span&gt;=&lt;/span&gt;&lt;span&gt; http_post(&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;https://bsky.social/xrpc/com.atproto.server.createSession&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;  &lt;/span&gt;&lt;/span&gt;&lt;span&gt;headers &lt;/span&gt;&lt;span&gt;=&gt;&lt;/span&gt;&lt;span&gt; MAP {&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;Content-Type&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;application/json&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;Accept&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;application/json&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;  &lt;/span&gt;&lt;/span&gt;&lt;span&gt;},&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;  &lt;/span&gt;&lt;/span&gt;&lt;span&gt;body &lt;/span&gt;&lt;span&gt;=&gt;&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;span&gt; c &lt;/span&gt;&lt;span&gt;FROM&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;./bluesky_credentials.json&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt; c)&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;) &lt;/span&gt;&lt;span&gt;-&lt;/span&gt;&lt;span&gt;&gt;&gt;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;body&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;-&lt;/span&gt;&lt;span&gt;&gt;&gt;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;accessJwt&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;WITH&lt;/span&gt;&lt;span&gt; topics &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;span&gt; col0 &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; topic, col1 &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; query_string &lt;/span&gt;&lt;span&gt;FROM&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;VALUES&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;DuckDB&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;,            &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;duckdb&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;),&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;Data Engineering&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;,  &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;&quot;data-engineering&quot; &quot;data engineering&quot; &quot;dataengineering&quot;&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;),&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;#databs&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;,           &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;#databs&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;),&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;  &lt;/span&gt;&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;),&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;topics_with_ts &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;topic,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;query_string,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;coalesce&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;max&lt;/span&gt;&lt;span&gt;(loaded_at), (&lt;/span&gt;&lt;span&gt;now&lt;/span&gt;&lt;span&gt;() &lt;/span&gt;&lt;span&gt;-&lt;/span&gt;&lt;span&gt; INTERVAL &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;30 days&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;)::&lt;/span&gt;&lt;span&gt;TIMESTAMP&lt;/span&gt;&lt;span&gt;) &lt;/span&gt;&lt;span&gt;as&lt;/span&gt;&lt;span&gt; last_loaded_at,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;FROM&lt;/span&gt;&lt;span&gt; topics &lt;/span&gt;&lt;span&gt;LEFT JOIN&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;bsky&lt;/span&gt;&lt;span&gt;.&lt;/span&gt;&lt;span&gt;posts&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;USING&lt;/span&gt;&lt;span&gt;(topic)&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;GROUP BY&lt;/span&gt;&lt;span&gt; ALL&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;),&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;json_posts &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;topic,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;(http_get(&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;      &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;https://bsky.social/xrpc/app.bsky.feed.searchPosts&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;      &lt;/span&gt;&lt;/span&gt;&lt;span&gt;headers &lt;/span&gt;&lt;span&gt;=&gt;&lt;/span&gt;&lt;span&gt; MAP {&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;        &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;Accept&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;application/json&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;        &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;Authorization&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;concat&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;Bearer &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;, getvariable(&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;access_jwt&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;)),&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;      &lt;/span&gt;&lt;/span&gt;&lt;span&gt;},&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;      &lt;/span&gt;&lt;/span&gt;&lt;span&gt;params &lt;/span&gt;&lt;span&gt;=&gt;&lt;/span&gt;&lt;span&gt; MAP {&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;        &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;q&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;: query_string,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;        &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;limit&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;: &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;100&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;        &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;since&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;: strftime(last_loaded_at, &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;%Y-%m-%dT%H:%M:%SZ&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;),&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;      &lt;/span&gt;&lt;/span&gt;&lt;span&gt;}&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;) &lt;/span&gt;&lt;span&gt;-&lt;/span&gt;&lt;span&gt;&gt;&gt;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;body&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;-&lt;/span&gt;&lt;span&gt;&gt;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;$.posts[*]&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;).unnest() &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; p&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;FROM&lt;/span&gt;&lt;span&gt; topics_with_ts&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;)&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;INSERT INTO&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;bsky&lt;/span&gt;&lt;span&gt;.&lt;/span&gt;&lt;span&gt;posts&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;BY&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;NAME&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;topic,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;(p &lt;/span&gt;&lt;span&gt;-&lt;/span&gt;&lt;span&gt;&gt;&gt;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;$.record.createdAt&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;)::&lt;/span&gt;&lt;span&gt;TIMESTAMP&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; created_at,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;p &lt;/span&gt;&lt;span&gt;-&lt;/span&gt;&lt;span&gt;&gt;&gt;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;cid&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; cid,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;p &lt;/span&gt;&lt;span&gt;-&lt;/span&gt;&lt;span&gt;&gt;&gt;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;$.author.handle&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; author_handle,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;concat&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;https://bsky.app/profile/&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;, author_handle, &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;/post/&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;, split_part(p &lt;/span&gt;&lt;span&gt;-&lt;/span&gt;&lt;span&gt;&gt;&gt;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;uri&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;, &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;/&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;, &lt;/span&gt;&lt;span&gt;-&lt;/span&gt;&lt;span&gt;1&lt;/span&gt;&lt;span&gt;)) &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;url&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;p &lt;/span&gt;&lt;span&gt;-&lt;/span&gt;&lt;span&gt;&gt;&gt;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;$.record.text&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;text&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;(p &lt;/span&gt;&lt;span&gt;-&lt;/span&gt;&lt;span&gt;&gt;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;likeCount&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;)::&lt;/span&gt;&lt;span&gt;INT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; like_count,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;(p &lt;/span&gt;&lt;span&gt;-&lt;/span&gt;&lt;span&gt;&gt;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;replyCount&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;)::&lt;/span&gt;&lt;span&gt;INT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; reply_count,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;(p &lt;/span&gt;&lt;span&gt;-&lt;/span&gt;&lt;span&gt;&gt;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;quoteCount&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;)::&lt;/span&gt;&lt;span&gt;INT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; quote_count,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;(p &lt;/span&gt;&lt;span&gt;-&lt;/span&gt;&lt;span&gt;&gt;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;repostCount&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;)::&lt;/span&gt;&lt;span&gt;INT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; repost_count,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;FROM&lt;/span&gt;&lt;span&gt; json_posts&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;);&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
The task is configured to run every hour and fetch new posts for the topics “DuckDB”, “Data Engineering”, and “#databs”.
&lt;br&gt;
Replace the topics with your own topics.
&lt;br&gt;
Then click &lt;strong&gt;Run&lt;/strong&gt; to try out the task. If the task runs successfully, click &lt;strong&gt;Create&lt;/strong&gt; and save it as &lt;code dir=&quot;auto&quot;&gt;Fetch Bluesky Posts&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;With the first data loaded, we can now create a dashboard to visualize the data.
&lt;br&gt;
Click on &lt;strong&gt;New&lt;/strong&gt; again and paste in the following SQL code:
&lt;br&gt;
&lt;br&gt;
&lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;Bluesky Analytics&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;::SECTION;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;min&lt;/span&gt;&lt;span&gt;(created_at)::&lt;/span&gt;&lt;span&gt;DATE&lt;/span&gt;&lt;span&gt;::DATEPICKER_FROM &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;start_date&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;max&lt;/span&gt;&lt;span&gt;(created_at)::&lt;/span&gt;&lt;span&gt;DATE&lt;/span&gt;&lt;span&gt;::DATEPICKER_TO &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; end_date,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;FROM&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;bsky&lt;/span&gt;&lt;span&gt;.&lt;/span&gt;&lt;span&gt;posts&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;Topics&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;::LABEL;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SELECT distinct&lt;/span&gt;&lt;span&gt; topic::DROPDOWN_MULTI &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; topics &lt;/span&gt;&lt;span&gt;FROM&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;bsky&lt;/span&gt;&lt;span&gt;.&lt;/span&gt;&lt;span&gt;posts&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;CREATE&lt;/span&gt;&lt;span&gt; TEMP VIEW posts &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;FROM&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;bsky&lt;/span&gt;&lt;span&gt;.&lt;/span&gt;&lt;span&gt;posts&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;WHERE&lt;/span&gt;&lt;span&gt; topic &lt;/span&gt;&lt;span&gt;in&lt;/span&gt;&lt;span&gt; getvariable(&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;topics&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;AND&lt;/span&gt;&lt;span&gt; created_at &lt;/span&gt;&lt;span&gt;BETWEEN&lt;/span&gt;&lt;span&gt; getvariable(&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;start_date&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;                       &lt;/span&gt;&lt;span&gt;AND&lt;/span&gt;&lt;span&gt; getvariable(&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;end_date&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;)&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;);&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;concat&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;bluesky_posts_&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;, today())::DOWNLOAD_CSV &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; CSV;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;*&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;FROM&lt;/span&gt;&lt;span&gt; posts;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;count&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;distinct&lt;/span&gt;&lt;span&gt; cid) &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;Total Posts Overall&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;FROM&lt;/span&gt;&lt;span&gt; posts;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;count&lt;/span&gt;&lt;span&gt;() &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;Total Posts&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;  &lt;/span&gt;&lt;/span&gt;&lt;span&gt;topic &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; Topic,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;FROM&lt;/span&gt;&lt;span&gt; posts &lt;/span&gt;&lt;span&gt;GROUP BY&lt;/span&gt;&lt;span&gt; topic &lt;/span&gt;&lt;span&gt;ORDER BY&lt;/span&gt;&lt;span&gt; ALL &lt;/span&gt;&lt;span&gt;DESC&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;Posts per Day&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;::LABEL;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;  &lt;/span&gt;&lt;/span&gt;&lt;span&gt;topic::CATEGORY,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;  &lt;/span&gt;&lt;/span&gt;&lt;span&gt;date_trunc(&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;day&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;, created_at)::XAXIS,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;count&lt;/span&gt;&lt;span&gt;()::BARCHART_STACKED,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;FROM&lt;/span&gt;&lt;span&gt; posts &lt;/span&gt;&lt;span&gt;GROUP BY&lt;/span&gt;&lt;span&gt; ALL &lt;/span&gt;&lt;span&gt;ORDER BY&lt;/span&gt;&lt;span&gt; ALL;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&apos;&lt;/span&gt;&lt;span&gt;::SECTION;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;Top Posters&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;::LABEL;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;FROM&lt;/span&gt;&lt;span&gt; (&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;    &lt;/span&gt;&lt;span&gt;count&lt;/span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;distinct&lt;/span&gt;&lt;span&gt; cid)::BARCHART &lt;/span&gt;&lt;span&gt;AS&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt;Total Posts&lt;/span&gt;&lt;span&gt;&quot;&lt;/span&gt;&lt;span&gt;,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;    &lt;/span&gt;&lt;/span&gt;&lt;span&gt;author_handle::YAXIS,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;FROM&lt;/span&gt;&lt;span&gt; posts &lt;/span&gt;&lt;span&gt;GROUP BY&lt;/span&gt;&lt;span&gt; ALL &lt;/span&gt;&lt;span&gt;ORDER BY&lt;/span&gt;&lt;span&gt; ALL &lt;/span&gt;&lt;span&gt;DESC&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;LIMIT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;10&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;) &lt;/span&gt;&lt;span&gt;ORDER BY&lt;/span&gt;&lt;span&gt; ALL;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;
&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;Likes By Time of Day&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;::LABEL;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;SELECT&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;  &lt;/span&gt;&lt;/span&gt;&lt;span&gt;topic::CATEGORY,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;  &lt;/span&gt;&lt;/span&gt;&lt;span&gt;date_trunc(&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;hour&lt;/span&gt;&lt;span&gt;&apos;&lt;/span&gt;&lt;span&gt;, created_at)::&lt;/span&gt;&lt;span&gt;TIME&lt;/span&gt;&lt;span&gt;::XAXIS,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;  &lt;/span&gt;&lt;span&gt;sum&lt;/span&gt;&lt;span&gt;(like_count)::BARCHART_STACKED,&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;FROM&lt;/span&gt;&lt;span&gt; posts &lt;/span&gt;&lt;span&gt;GROUP BY&lt;/span&gt;&lt;span&gt; ALL &lt;/span&gt;&lt;span&gt;ORDER BY&lt;/span&gt;&lt;span&gt; ALL;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
Now click &lt;strong&gt;Create&lt;/strong&gt; and save the dashboard as &lt;code dir=&quot;auto&quot;&gt;Bluesky Analytics&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Click on &lt;strong&gt;View Dashboard&lt;/strong&gt; in the top right corner to have a better look at the whole dashboard.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;And you are done!
Please reach out, ask questions and I would love to see what you built.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Shaper is open source and free to use. It’s simple to run on your own server and so you can easily share dashboards with others. Find out more on Github:&lt;/em&gt;&lt;/p&gt;
&lt;div&gt;&lt;a href=&quot;https://github.com/taleshape-com/shaper&quot; target=&quot;_blank&quot;&gt;  taleshape-com/shaper &lt;svg aria-hidden=&quot;true&quot; width=&quot;16&quot; height=&quot;16&quot; viewBox=&quot;0 0 24 24&quot; fill=&quot;currentColor&quot;&gt;&lt;path d=&quot;M12 .3a12 12 0 0 0-3.8 23.38c.6.12.83-.26.83-.57L9 21.07c-3.34.72-4.04-1.61-4.04-1.61-.55-1.39-1.34-1.76-1.34-1.76-1.08-.74.09-.73.09-.73 1.2.09 1.83 1.24 1.83 1.24 1.08 1.83 2.81 1.3 3.5 1 .1-.78.42-1.31.76-1.61-2.67-.3-5.47-1.33-5.47-5.93 0-1.31.47-2.38 1.24-3.22-.14-.3-.54-1.52.1-3.18 0 0 1-.32 3.3 1.23a11.5 11.5 0 0 1 6 0c2.28-1.55 3.29-1.23 3.29-1.23.64 1.66.24 2.88.12 3.18a4.65 4.65 0 0 1 1.23 3.22c0 4.61-2.8 5.63-5.48 5.92.42.36.81 1.1.81 2.22l-.01 3.29c0 .31.2.69.82.57A12 12 0 0 0 12 .3Z&quot;&gt;&lt;/path&gt;&lt;/svg&gt; &lt;/a&gt; &lt;/div&gt;
&lt;div&gt; &lt;h2&gt;Subscribe to Newsletter&lt;/h2&gt; &lt;p&gt;Get the new post delivered straight to your inbox.&lt;/p&gt; &lt;form id=&quot;blog-signup-form&quot; action=&quot;https://app.loops.so/api/newsletter-form/cmciwkod00u4a0iy59t1zcyk4&quot;&gt; &lt;div&gt; &lt;input id=&quot;blog-email&quot; type=&quot;email&quot; name=&quot;email&quot; required placeholder=&quot;Your email address&quot; autocomplete=&quot;email&quot;&gt; &lt;p&gt;&lt;/p&gt; &lt;/div&gt; &lt;button type=&quot;submit&quot;&gt;
Subscribe
&lt;/button&gt; &lt;/form&gt; &lt;p&gt;
Thank you for signing up!
&lt;/p&gt; &lt;p&gt;Unsubscribe at any time.&lt;/p&gt; &lt;/div&gt;  </content:encoded></item><item><title>Turn Your DuckDB Projects Into Interactive Dashboards</title><link>https://taleshape.com/blog/turn-your-duckdb-projects-into-interactive-dashboards/</link><guid isPermaLink="true">https://taleshape.com/blog/turn-your-duckdb-projects-into-interactive-dashboards/</guid><pubDate>Tue, 26 Aug 2025 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;DuckDB is awesome and it’s a great tool to explore and transform data.
But DuckDB doesn’t help you visualize and share data with others.&lt;/p&gt;
&lt;p&gt;That’s where Shaper comes in.&lt;/p&gt;
&lt;p&gt;With Shaper you can build interactive dashboards completely in SQL.&lt;/p&gt;
&lt;p&gt;Shaper is built on top of DuckDB and works with all your existing data and queries.&lt;/p&gt;
&lt;p&gt;Running Shaper on your laptop is as easy as running a single command:&lt;/p&gt;
&lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;span&gt;&lt;/span&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;npx&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;shaper&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;p&gt;Then open &lt;a href=&quot;http://localhost:5454/new&quot;&gt;http://localhost:5454/new&lt;/a&gt; in your browser:&lt;/p&gt;
&lt;img src=&quot;https://taleshape.com/_astro/new-dashboard-page.DZVayEoo_Z15N8iI.webp&quot; alt=&quot;New Dashboard view&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; fetchpriority=&quot;auto&quot; width=&quot;1440&quot; height=&quot;718&quot;&gt;
&lt;p&gt;And running Shaper on a server is just as simple:&lt;/p&gt;
&lt;div&gt; &lt;span&gt; &lt;a href=&quot;https://taleshape.com/shaper/docs/deploy-to-production&quot;&gt; &lt;span&gt;Deploy To Production&lt;/span&gt; &lt;/a&gt;  &lt;/span&gt; &lt;svg aria-hidden=&quot;true&quot; width=&quot;16&quot; height=&quot;16&quot; viewBox=&quot;0 0 24 24&quot; fill=&quot;currentColor&quot;&gt;&lt;path d=&quot;M17.92 11.62a1.001 1.001 0 0 0-.21-.33l-5-5a1.003 1.003 0 1 0-1.42 1.42l3.3 3.29H7a1 1 0 0 0 0 2h7.59l-3.3 3.29a1.002 1.002 0 0 0 .325 1.639 1 1 0 0 0 1.095-.219l5-5a1 1 0 0 0 .21-.33 1 1 0 0 0 0-.76Z&quot;&gt;&lt;/path&gt;&lt;/svg&gt; &lt;/div&gt; 
&lt;p&gt;You can then connect Shaper directly to your &lt;a href=&quot;https://taleshape.com/shaper/docs/connecting-to-data-sources/&quot;&gt;production data&lt;/a&gt; and share dashboards either with simple &lt;a href=&quot;https://taleshape.com/shaper/docs/public-sharing/&quot;&gt;links&lt;/a&gt; or by &lt;a href=&quot;https://taleshape.com/shaper/docs/dashboard-embedding&quot;&gt;embedding&lt;/a&gt; dashboards directly into your application.&lt;/p&gt;
&lt;p&gt;It’s all open source and free to use. So why not give it a try?&lt;/p&gt;
&lt;div&gt;&lt;a href=&quot;https://taleshape.com/shaper/docs/getting-started&quot;&gt;  Get Started  &lt;/a&gt; &lt;/div&gt;
&lt;div&gt; &lt;h2&gt;Subscribe to Newsletter&lt;/h2&gt; &lt;p&gt;Get the new post delivered straight to your inbox.&lt;/p&gt; &lt;form id=&quot;blog-signup-form&quot; action=&quot;https://app.loops.so/api/newsletter-form/cmciwkod00u4a0iy59t1zcyk4&quot;&gt; &lt;div&gt; &lt;input id=&quot;blog-email&quot; type=&quot;email&quot; name=&quot;email&quot; required placeholder=&quot;Your email address&quot; autocomplete=&quot;email&quot;&gt; &lt;p&gt;&lt;/p&gt; &lt;/div&gt; &lt;button type=&quot;submit&quot;&gt;
Subscribe
&lt;/button&gt; &lt;/form&gt; &lt;p&gt;
Thank you for signing up!
&lt;/p&gt; &lt;p&gt;Unsubscribe at any time.&lt;/p&gt; &lt;/div&gt;  </content:encoded></item><item><title>Why I am excited about Shaper&apos;s new Tasks feature</title><link>https://taleshape.com/blog/why-i-am-excited-about-shapers-new-task-feature/</link><guid isPermaLink="true">https://taleshape.com/blog/why-i-am-excited-about-shapers-new-task-feature/</guid><pubDate>Mon, 18 Aug 2025 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;I just shipped the biggest update since I started building Shaper.&lt;/p&gt;
&lt;p&gt;Tasks let you automate data workflows right within Shaper.&lt;/p&gt;
&lt;p&gt;A few examples:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Load the latest data from a database or API&lt;/li&gt;
&lt;li&gt;Transform data to speed up dashboard queries&lt;/li&gt;
&lt;li&gt;Archive old data to S3&lt;/li&gt;
&lt;li&gt;Send a notification to Slack for critical data insights&lt;/li&gt;
&lt;li&gt;Email a monthly Excel report to your customers&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;With the power of DuckDB and its extensions, you can do all this and more with simple SQL queries.&lt;/p&gt;
&lt;div&gt;&lt;h2 id=&quot;the-missing-piece&quot;&gt;The Missing Piece&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;Since the beginning I wanted Shaper to be a complete data platform in a single tool - From ingesting and storing data to visualizing and sharing it.&lt;/p&gt;
&lt;p&gt;The Tasks feature is the missing piece in the middle - Transform data to get it into the shape that you can actually visualize and share it.&lt;/p&gt;
&lt;div&gt;&lt;h2 id=&quot;growing-with-your-data-needs&quot;&gt;Growing With Your Data Needs&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;You can think of tasks as CRON jobs with more flexible scheduling and integrated into the platform.
But there is a lot of things Tasks don’t do and likely never will. Shaper’s goal is simplicity when starting out with data projects.
And once your data needs become more complex you can introduce a dedicated data processing stack to complement Shaper.&lt;/p&gt;
&lt;p&gt;Find all the details in the &lt;a href=&quot;https://taleshape.com/shaper/docs/tasks-and-scheduling&quot;&gt;docs&lt;/a&gt;.&lt;/p&gt;
&lt;div&gt; &lt;h2&gt;Subscribe to Newsletter&lt;/h2&gt; &lt;p&gt;Get the new post delivered straight to your inbox.&lt;/p&gt; &lt;form id=&quot;blog-signup-form&quot; action=&quot;https://app.loops.so/api/newsletter-form/cmciwkod00u4a0iy59t1zcyk4&quot;&gt; &lt;div&gt; &lt;input id=&quot;blog-email&quot; type=&quot;email&quot; name=&quot;email&quot; required placeholder=&quot;Your email address&quot; autocomplete=&quot;email&quot;&gt; &lt;p&gt;&lt;/p&gt; &lt;/div&gt; &lt;button type=&quot;submit&quot;&gt;
Subscribe
&lt;/button&gt; &lt;/form&gt; &lt;p&gt;
Thank you for signing up!
&lt;/p&gt; &lt;p&gt;Unsubscribe at any time.&lt;/p&gt; &lt;/div&gt;  </content:encoded></item><item><title>Shaper Now Open Source</title><link>https://taleshape.com/blog/shaper-now-open-source/</link><guid isPermaLink="true">https://taleshape.com/blog/shaper-now-open-source/</guid><pubDate>Tue, 05 Aug 2025 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;Shaper is a minimal data platform for embedded analytics.
It allows you to build interactive data dashboards and embed them into your web application.&lt;/p&gt;
&lt;p&gt;And starting today, Shaper is open source:&lt;/p&gt;
&lt;div&gt;&lt;a href=&quot;https://github.com/taleshape-com/shaper&quot; target=&quot;_blank&quot;&gt;  taleshape-com/shaper &lt;svg aria-hidden=&quot;true&quot; width=&quot;16&quot; height=&quot;16&quot; viewBox=&quot;0 0 24 24&quot; fill=&quot;currentColor&quot;&gt;&lt;path d=&quot;M12 .3a12 12 0 0 0-3.8 23.38c.6.12.83-.26.83-.57L9 21.07c-3.34.72-4.04-1.61-4.04-1.61-.55-1.39-1.34-1.76-1.34-1.76-1.08-.74.09-.73.09-.73 1.2.09 1.83 1.24 1.83 1.24 1.08 1.83 2.81 1.3 3.5 1 .1-.78.42-1.31.76-1.61-2.67-.3-5.47-1.33-5.47-5.93 0-1.31.47-2.38 1.24-3.22-.14-.3-.54-1.52.1-3.18 0 0 1-.32 3.3 1.23a11.5 11.5 0 0 1 6 0c2.28-1.55 3.29-1.23 3.29-1.23.64 1.66.24 2.88.12 3.18a4.65 4.65 0 0 1 1.23 3.22c0 4.61-2.8 5.63-5.48 5.92.42.36.81 1.1.81 2.22l-.01 3.29c0 .31.2.69.82.57A12 12 0 0 0 12 .3Z&quot;&gt;&lt;/path&gt;&lt;/svg&gt; &lt;/a&gt; &lt;/div&gt;
&lt;div&gt;&lt;h2 id=&quot;why-open-source&quot;&gt;Why Open Source?&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;My main motivation to open source Shaper is to make it accessible to as many users as possible.&lt;/p&gt;
&lt;p&gt;To get the most out of Shaper you want to integrate it deeply into your product and infrastructure.
Deeply integrating software from a 3rd-party company always comes with risks of being too dependent on another company.
With Shaper you are running open source software in your own infrastructure.&lt;/p&gt;
&lt;p&gt;Using an open source solution also means you can verify security and privacy practices.
This combined with being able to run Shaper in your own infrastructure makes it a great choice for use cases that handle sensitive data.&lt;/p&gt;
&lt;p&gt;Lastly, Shaper itself is built on top of other amazing open source projects, including &lt;a href=&quot;https://duckdb.org/&quot;&gt;DuckDB&lt;/a&gt;, &lt;a href=&quot;https://echarts.apache.org/en/index.html&quot;&gt;ECharts&lt;/a&gt;, and &lt;a href=&quot;https://nats.io/&quot;&gt;NATS&lt;/a&gt;.
To open source Shaper also means paying forward the spirit that made Shaper possible in the first place.&lt;/p&gt;
&lt;div&gt;&lt;h2 id=&quot;shaper-pro&quot;&gt;Shaper PRO&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;One of my biggest motivations for building Shaper was to enable as many teams as possible to get value out of their data and make their data accessible to their users.&lt;/p&gt;
&lt;p&gt;Shaper is designed to be simple to run and easy to use. But it cannot remove the inherent complexity of any data project.&lt;/p&gt;
&lt;p&gt;That’s where &lt;strong&gt;Shaper PRO&lt;/strong&gt; comes in.&lt;/p&gt;
&lt;p&gt;You can think of it as hiring me as a part-time data engineer that manages your data platform and helps you implement your data use cases.
What you get:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Shaper, fully-managed: Monitoring, updates, backups, security, compliance, and high-availability deployments&lt;/li&gt;
&lt;li&gt;Extensive support: Integrate Shaper into your product, connect your data sources, manage data, build dashboards&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Please &lt;a href=&quot;mailto:jorin@taleshape.com&quot;&gt;reach out&lt;/a&gt; if this is something you are interested in.&lt;/p&gt;
&lt;div&gt;&lt;h2 id=&quot;give-shaper-a-try&quot;&gt;Give Shaper a try&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;Trying out Shaper is as easy as running a single command:&lt;/p&gt;
&lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;span&gt;&lt;/span&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;docker&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;run&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;--rm&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;-it&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;-p5454:5454&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;taleshape/shaper&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;p&gt;Then open &lt;a href=&quot;http://localhost:5454/new&quot;&gt;http://localhost:5454/new&lt;/a&gt; in your browser.&lt;/p&gt;
&lt;p&gt;I am curious to hear your thoughts and feedback. So don’t hesitate to open an issue or discussion on &lt;a href=&quot;https://github.com/taleshape-com/shaper&quot;&gt;Github&lt;/a&gt;. Or just send a message.&lt;/p&gt;
&lt;p&gt;Thank you!&lt;br&gt;
Jorin&lt;/p&gt;
&lt;div&gt; &lt;h2&gt;Subscribe to Newsletter&lt;/h2&gt; &lt;p&gt;Get the new post delivered straight to your inbox.&lt;/p&gt; &lt;form id=&quot;blog-signup-form&quot; action=&quot;https://app.loops.so/api/newsletter-form/cmciwkod00u4a0iy59t1zcyk4&quot;&gt; &lt;div&gt; &lt;input id=&quot;blog-email&quot; type=&quot;email&quot; name=&quot;email&quot; required placeholder=&quot;Your email address&quot; autocomplete=&quot;email&quot;&gt; &lt;p&gt;&lt;/p&gt; &lt;/div&gt; &lt;button type=&quot;submit&quot;&gt;
Subscribe
&lt;/button&gt; &lt;/form&gt; &lt;p&gt;
Thank you for signing up!
&lt;/p&gt; &lt;p&gt;Unsubscribe at any time.&lt;/p&gt; &lt;/div&gt;  </content:encoded></item><item><title>What&apos;s Embedded Analytics?</title><link>https://taleshape.com/blog/what-is-embedded-analytics/</link><guid isPermaLink="true">https://taleshape.com/blog/what-is-embedded-analytics/</guid><pubDate>Fri, 25 Jul 2025 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;Hi, my name is Jorin and I am developing Shaper — A Minimal Solution for Embedded Analytics.&lt;/p&gt;
&lt;p&gt;But what does &lt;strong&gt;Embedded Analytics&lt;/strong&gt; even mean?&lt;/p&gt;
&lt;p&gt;Are your users asking for analytics features to get access to their data?
With embedded analytics your users get what they are asking for without you having to build this functionality from scratch.&lt;/p&gt;
&lt;p&gt;Let’s see what this can look like with Shaper!&lt;/p&gt;
&lt;p&gt;The following is a live example of a dashboard embedded in this website. Go ahead, filter the data or download it as a CSV file:&lt;/p&gt;
&lt;div data-dashboard-id=&quot;vmw80z157nz7200teo9l35yi&quot;&gt;&lt;/div&gt; 
&lt;p&gt;We created this dashboard using the Shaper editor:&lt;/p&gt;
&lt;img src=&quot;https://taleshape.com/_astro/embedding-demo-editor-view.m3zLdrEy_Z1sHVfw.webp&quot; alt=&quot;Editor Screenshot&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot; fetchpriority=&quot;auto&quot; width=&quot;1056&quot; height=&quot;524&quot;&gt;
&lt;p&gt;Then we embed the dashboard into our web application using JavaScript:&lt;/p&gt;
&lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;shaper&lt;/span&gt;&lt;span&gt;.&lt;/span&gt;&lt;span&gt;dashboard&lt;/span&gt;&lt;span&gt;&lt;span&gt;({ &lt;/span&gt;&lt;span&gt;container&lt;/span&gt;&lt;span&gt;, &lt;/span&gt;&lt;span&gt;dashboardId&lt;/span&gt;&lt;span&gt;, &lt;/span&gt;&lt;span&gt;getJwt&lt;/span&gt;&lt;span&gt; })&lt;/span&gt;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;p&gt;We customize the look of the dashboards to fit our UI:&lt;/p&gt;
&lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;--shaper-font: DMSans;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;--shaper-primary-color: &lt;/span&gt;&lt;span&gt;#a5cde6;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;--shaper-text-color: &lt;/span&gt;&lt;span&gt;#3d3f45&lt;/span&gt;&lt;span&gt;;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;/* ... */&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;p&gt;And behind the scenes we can generate a JSON Web Token with permissions for the logged-in user to make sure they only see what they are allowed to:&lt;/p&gt;
&lt;div&gt;&lt;figure&gt;&lt;figcaption&gt;&lt;/figcaption&gt;&lt;pre&gt;&lt;code&gt;&lt;div&gt;&lt;div&gt;&lt;span&gt;&lt;span&gt;{ &lt;/span&gt;&lt;span&gt;jwt&lt;/span&gt;&lt;span&gt; } &lt;/span&gt;&lt;/span&gt;&lt;span&gt;=&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;await&lt;/span&gt;&lt;span&gt; &lt;/span&gt;&lt;span&gt;POST&lt;/span&gt;&lt;span&gt;&lt;span&gt;(&lt;/span&gt;&lt;span&gt;api&lt;/span&gt;&lt;span&gt;, { &lt;/span&gt;&lt;span&gt;token&lt;/span&gt;&lt;span&gt;, &lt;/span&gt;&lt;span&gt;dashboardId&lt;/span&gt;&lt;span&gt;, variables: { &lt;/span&gt;&lt;span&gt;user_id&lt;/span&gt;&lt;span&gt; } })&lt;/span&gt;&lt;/span&gt;&lt;/div&gt;&lt;/div&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/figure&gt;&lt;/div&gt;
&lt;p&gt;Obviously, the above example is publicly accessible on the internet, so we didn’t have to authenticate the user.&lt;/p&gt;
&lt;div&gt;&lt;h2 id=&quot;why-shaper&quot;&gt;Why Shaper?&lt;/h2&gt;&lt;/div&gt;
&lt;p&gt;Building customized and deeply integrated data dashboards can get complicated — especially once users start coming back to you with new questions.&lt;/p&gt;
&lt;p&gt;My goal with Shaper is to make embedded analytics so simple that you don’t need a dedicated data team or pay a vendor to solve it for you.&lt;/p&gt;
&lt;p&gt;Give it a try and let me know what you think.&lt;/p&gt;
&lt;div&gt;&lt;a href=&quot;https://taleshape.com/shaper/docs/getting-started/&quot;&gt;  Get Started &lt;svg aria-hidden=&quot;true&quot; width=&quot;16&quot; height=&quot;16&quot; viewBox=&quot;0 0 24 24&quot; fill=&quot;currentColor&quot;&gt;&lt;path fill-rule=&quot;evenodd&quot; d=&quot;M1.44 8.855v-.001l3.527-3.516c.34-.344.802-.541 1.285-.548h6.649l.947-.947c3.07-3.07 6.207-3.072 7.62-2.868a1.821 1.821 0 0 1 1.557 1.557c.204 1.413.203 4.55-2.868 7.62l-.946.946v6.649a1.845 1.845 0 0 1-.549 1.286l-3.516 3.528a1.844 1.844 0 0 1-3.11-.944l-.858-4.275-4.52-4.52-2.31-.463-1.964-.394A1.847 1.847 0 0 1 .98 10.693a1.843 1.843 0 0 1 .46-1.838Zm5.379 2.017-3.873-.776L6.32 6.733h4.638l-4.14 4.14Zm8.403-5.655c2.459-2.46 4.856-2.463 5.89-2.33.134 1.035.13 3.432-2.329 5.891l-6.71 6.71-3.561-3.56 6.71-6.711Zm-1.318 15.837-.776-3.873 4.14-4.14v4.639l-3.364 3.374Z&quot; clip-rule=&quot;evenodd&quot;&gt;&lt;/path&gt;&lt;path d=&quot;M9.318 18.345a.972.972 0 0 0-1.86-.561c-.482 1.435-1.687 2.204-2.934 2.619a8.22 8.22 0 0 1-1.23.302c.062-.365.157-.79.303-1.229.415-1.247 1.184-2.452 2.62-2.935a.971.971 0 1 0-.62-1.842c-.12.04-.236.084-.35.13-2.02.828-3.012 2.588-3.493 4.033a10.383 10.383 0 0 0-.51 2.845l-.001.016v.063c0 .536.434.972.97.972H2.24a7.21 7.21 0 0 0 .878-.065c.527-.063 1.248-.19 2.02-.447 1.445-.48 3.205-1.472 4.033-3.494a5.828 5.828 0 0 0 .147-.407Z&quot;&gt;&lt;/path&gt;&lt;/svg&gt;  &lt;/a&gt; &lt;/div&gt;
&lt;div&gt; &lt;h2&gt;Subscribe to Newsletter&lt;/h2&gt; &lt;p&gt;Get the new post delivered straight to your inbox.&lt;/p&gt; &lt;form id=&quot;blog-signup-form&quot; action=&quot;https://app.loops.so/api/newsletter-form/cmciwkod00u4a0iy59t1zcyk4&quot;&gt; &lt;div&gt; &lt;input id=&quot;blog-email&quot; type=&quot;email&quot; name=&quot;email&quot; required placeholder=&quot;Your email address&quot; autocomplete=&quot;email&quot;&gt; &lt;p&gt;&lt;/p&gt; &lt;/div&gt; &lt;button type=&quot;submit&quot;&gt;
Subscribe
&lt;/button&gt; &lt;/form&gt; &lt;p&gt;
Thank you for signing up!
&lt;/p&gt; &lt;p&gt;Unsubscribe at any time.&lt;/p&gt; &lt;/div&gt;  </content:encoded></item></channel></rss>