It was observed that the next-campaign-subscribers query on an instance
with ~9 million subscribers had slowed down significantly. Fetching
a batch of 5k subscribers was taking around ~25 seconds.
After multiple hours of debugging and trial and errors, it turned out
that Postgres was doing very poor query planning on JOINs with CTEs
because of the dynamic cardinality of some CTEs (even with just 1 row).
Afer rewriting the query and adding a hack to overcome the CTE
cardinality issue, the same query now takes a few milliseconds,
a speed up of several orders of magnitude.
- Change `query-lists` query to aggregate the subscriber count by
status (confirmed, unsubscribed etc.) and expose them under a new
`subscriber_statuses: {}` field in the `GET /lists` API.
- Display the statuses and counts in the lists table on the UI.
Closes#616
The analytics page showed non-unique counts for views and clicks which
was misleading and source of confusion: #522, #561, #571, #676, #680
This commit changes this behaviour to pull unique views and clicks when
individual subscriber tracking is turned on in settings, and non-unique
counts when it is turned off (as `subscriber_id` in `campaign_views`
and `link_clicks` will be NULL, rendering unique queries dysfunctional).
This commit changes the stats SQL queries to use string interpolation
to either to SELECT `*` or `DISTINCT subscriber_id` on app boot based
on the setting in the DB. This involves significant changes to how
queries are read and prepared on init.
- Refactor `initQueries()` to `readQueries()` and `prepareQueries()`.
- Read queries first before preparing.
- Load settings from the DB using the read settings query.
- Prepare queries next. Use the privacy setting from the DB to apply
string interpolation to the analytics queries to pull
unique/non-unique before preparing the queries.
On the UI:
- Show a note on the analytics page about unique/non-unique counts.
- Hide the % donut charts on the analytics page in non-unique mode.
Closes#676, closes#680
- Add new `headers[]` column to the campain table.
- Add new headers box to the campaign UI that takes a JSON array of
custom headers like the headers on the SMTP settings UI.
- Headers are added to e-mails and messenger postback webhooks.
- Add cypress tests.
Closes#514.
- Add route /lists/:id URI to load subscriber edit modal.
- Make list name open the edit popup to be consistent with all other
table views.
- Refactor get-lists query to make single list look up faster.
- Add indexes.
- Refactor dashboard charts and view/click count queries.
(~10x speed bump on a setup of 7mn subscribers and 80mn views)
- Refactor get subscriber queries.
(~10x speed bump on 7mn subscribers)
- Make subscriber UI issue an equality query for email seach strings.
- Blocklist or unsubscribe subscribers based on a bounce threshold
- Add /bounces UI for viewing bounces and in the subscriber view
- Add settings UI for managing bounce settings
- Add support for scanning POP3 bounce mailboxes
- Add a generic webhook for posting custom bounces at /webhooks/bounce
- Add SES bounce webhook support at /webhooks/services/ses
- Add Sendgrid bounce webhook support at /webhooks/services/sendgrid
- Refactor subimporter New*() funcs to take opt structs.
- Refactor and simplify Vue code.
- Remove redundant i18n entries and use existing ones.
- Remove redundant subimporter constants and use existing ones.
- Consider 'overwrite' option for subscription status as well.
- Write Cypress integration tests for the new feature.
Sending th optional flag as `trunue` in the POST /api/subscrirs
body will skip sending opt-iconfirmation e-mails to subscribers
and mark list subscriptions in the request a`confirmed`.
Change the behaviour where not passing attribs to the update API
overwrites the attribs with empty values. This commit changes the
behaviour so that in the absence of the attribs field in the
subscriber API, the existing value in the DB is retained.
If a user is already subscribed to an optin list but hasn't
confirmed, subscribing using the same e-mail id from the public
form now re-sends the optin e-mail while also showing an
appropriate message on the frontend rather than just saying
"subscribed successfully".
https://github.com/knadh/listmonk/issues/266https://github.com/knadh/listmonk/issues/264
In addition to generating HTML forms for selected public lists,
the form page now shows a URL (/subscription/form) that can be
publicly shared to solicit subscriptions. The page lists all
public lists in the database. This page can be disabled on the
Settings UI.
This commit removes the Go html2text lib that would automatically
convert all HTML messages to plaintext and add them as the alt
text body to outgoing e-mails. This lib also had memory leak
issues with certain kinds of HTML templates.
A new UI field for optionally adding an alt plaintext body to
a campaign is added. On enabling, it converts the HTML message in
the campaign editor into plaintext (using the textversionjs lib).
This introduces breaking changes in the campaigns table schema,
model, and template compilation.
Lists, campaigns, and subscribers tables now support server-side
sorting from the UI. This significantly changes the internal
queries from prepared to string interpolated to support dynamic
sort params.
The link_clicks.link_id table was NULLable incorrectly. Links that
do not exist should not register a tracking entry. Fix the query
and also update the schema + migration (breaking table change).
A new toggle switch in Settings -> Privacy, which is off by
default, allows campaign views (pixel) and link clicks to function
without registering the subscriber ID against view and click
events, anonymising tracking. When off, the subscriber UUIDs in
view and link tracking URLs are removed, anonymising subscriber
information from HTTP logs as well.
This is a major feature that builds upon the `Messenger` interface
that has been in listmonk since its inception (with SMTP as the only
messenger). This commit introduces a new Messenger implementation, an
HTTP "postback", that can post campaign messages as a standard JSON
payload to arbitrary HTTP servers. These servers can in turn push them
to FCM, SMS, or any or any such upstream, enabling listmonk to be a
generic campaign messenger for any type of communication, not just
e-mails.
Postback HTTP endpoints can be defined in settings and they can be
selected on campaigns.
installations.
In v0.7.0's automatic DB migration, the default value 'enabled'
of subscribers.status was missed incorrectly, causing imports to
break on auto-upgraded versions. This is now fixed by explictly
inserting the status field in CSV imports.
- Add missing `app.root_url` key in migration.
- Register `/settings` handler in the backend.
- Add dummy dots in secret fields on the UI for visibility.
This is a major breaking change that moves away from having the
entire app configuration in external TOML files to settings being
in the database with a UI to update them dynamically.
The app loads all config into memory (app settings, SMTP conf)
on boot. "Hot" replacing them is complex and it's a fair tradeoff
to instead just restart the application as it is practically
instant.
A new `settings` table stores arbitrary string keys with a JSONB
value field which happens to support arbitrary types. After every
settings update, the app gracefully releases all resources
(HTTP server, DB pool, SMTP pool etc.) and restarts itself,
occupying the same PID. If there are any running campaigns, the
auto-restart doesn't happen and the user is prompted to invoke
it manually with a one-click button once all running campaigns
have been paused.
- Fix path related issues in filesystem and S3.
- Add checks for S3 "/" path prefix.
- Add support for custom S3 domain names.
- Remove obsolete `width` and `height` columns from media table (breaking)
- Add `provider` field to media table (breaking)
- antd+react was resulting in extremely clunky and unreadable
spaghetti frontend code (primarily due to how antd is).
- Buefy is lighter by an order of magnitude, has excellent
responsive views (especially tables) and usability.
- Vue's templating produces far more readable template code.
