The `rate` field `/api/campaigns/running/stats` returned was computed
based on the total time spent from the start of the campaign to the
current time. This meant that for large campaigns, if there were
pauses or slowdowns in between, the rate would be skewed heavily
making it useless to figure out the current send rate.
This commit introduces a realtime running rate counter in the campaign
manager that returns accurate (running) send rates for the last minute.
The `rate` field in the API now shows the live running rate and a
new `net_rate` field shows the rate from the beginning of the campaign.
- Change `query-lists` query to aggregate the subscriber count by
status (confirmed, unsubscribed etc.) and expose them under a new
`subscriber_statuses: {}` field in the `GET /lists` API.
- Display the statuses and counts in the lists table on the UI.
Closes#616
The analytics page showed non-unique counts for views and clicks which
was misleading and source of confusion: #522, #561, #571, #676, #680
This commit changes this behaviour to pull unique views and clicks when
individual subscriber tracking is turned on in settings, and non-unique
counts when it is turned off (as `subscriber_id` in `campaign_views`
and `link_clicks` will be NULL, rendering unique queries dysfunctional).
This commit changes the stats SQL queries to use string interpolation
to either to SELECT `*` or `DISTINCT subscriber_id` on app boot based
on the setting in the DB. This involves significant changes to how
queries are read and prepared on init.
- Refactor `initQueries()` to `readQueries()` and `prepareQueries()`.
- Read queries first before preparing.
- Load settings from the DB using the read settings query.
- Prepare queries next. Use the privacy setting from the DB to apply
string interpolation to the analytics queries to pull
unique/non-unique before preparing the queries.
On the UI:
- Show a note on the analytics page about unique/non-unique counts.
- Hide the % donut charts on the analytics page in non-unique mode.
Closes#676, closes#680
- Updating a subscriber no longer triggers an opt-in confirmation mail
as `POST /api/subscribers/:id/optin` allows that.
- A "Send opt-in confirmation" option is added to the subscriber
update UI.
Closes#656.
- Add new `headers[]` column to the campain table.
- Add new headers box to the campaign UI that takes a JSON array of
custom headers like the headers on the SMTP settings UI.
- Headers are added to e-mails and messenger postback webhooks.
- Add cypress tests.
Closes#514.
- Add support for TLS in `smtppool` (v0.4.0) and upgrade the lib.
- Change `tls_enabled: bool` in the settings table to string
`tls_type: STARTTLS|TLS|none` and on the settings UI.
- Add DB migrations and schema changes to apply the field change.
Closes#504.
This feature was originally authored by @sweetppro in PR #438.
However, since the PR ended up in an unclean state with
multiple master merges (instead of rebase) from the upstream, there are
several commits that are out of order and can can no longer be be
squashed for a clean feature merge.
This commit aggregates the changes from the original PR and applies the
following fixes on top of it.
- Add custom admin JS box to appearance UI.
- Refactor i18n language strings.
- Add handlers and migrations for the new `appearance.admin.custom_js`
field.
- Fix migration version to `v2.1.0`
- Load custom appearance CSS/JS bytes into global constants during boot
instead of making a DB call on every request.
- Fix and canonicalize URIs from `/api/custom*` to `/public/*.css`
and `/admin/*.css`. Add proxy paths to yarn proxy config.
- Remove redundant HTTP handlers for different custom appearance files
and refactor into a single handler `serveCustomApperance()`
- Fix content-type and UTF8 encoding headers for different file types.
- Fix incorrect registration of public facing custom CSS/JS handlers
in the authenticated admin URI group.
- Fix merge conflicts in `Settings.vue`.
- Minor HTML and style fixes.
- Remove the `AppearanceEditor` component and use the existing
`HTMLEditor` component instead.
- Add `language` prop to the `HTMLEditor` component.
Co-authored-by: SweetPPro <sweetppro@users.noreply.github.com>
- echo is now on v4 with major changes including a few breaking changes
- bind() behaviour is now strict. JSON / form etc. unmarshalling of
request data need appropriate `json`, `form` tags. Missing tags for
the public subscription page is added in this commit.
- This also closes#602.
Every listmonk instance scans the DB periodically to look for
running campaigns to process. This made running multiple instances of
listmonk impractical as they would all pick up the same running
campaign and process them, resulting in duplicate e-mails.
This commit adds a `--passive` flag to the binary that runs listmonk
in a "passive" mode where campaign processing is disabled. This allows
multiple instances of listmonk to be run to handle different kinds of
requests if there is a requirement (scale/traffic?). It is important
to note that there should only be one non-passive instance running at
any given time. If distributed campaign processing is ever considered,
this will change.
If `<!doctype html>` is not found in static/email-templates/base.html,
all system e-mail templates are assumed to be plaintext and go out
as content-type: plaintext e-mails. With this, all HTML tags can
be stripped out of the system e-mail templates (while maintaining
Go template tags and logic) to have plaintext system e-mail templates.
Closes#546
BasicAuth without an explicit landing page or a logout option has
sometimes been confusing to users. This commit adds a static
landing page on / with a login link and a logout option in the admin
that "logs out" BasicAuth session by posting invalid credentials to
the server to obtain a 401.
- Refactor codeflask HTML editor into a standalone html-editor
component.
- Replace the plaintext box in the template editor with html-editor.
- Replace codeflask in the campaign editor with the new html-editor.
- Refactor templates Cypress tests to test the new editor.
- Refactor campaigns Cypress tests to test the new editor and also
test switching between different editors and content formats.
The default `{{ TrackLink "https://listmonk.app" }}` template function
is clumsy to write and does breaks WYSIWYG editors and HTML syntax
highlighting because of the quotes. The new syntax doesn't break HTML
and is easier to write.
Eg: `<a href="https://listmonk.app@TrackLink">Link</a>`
- Introduce @TrackLink shorthand.
- Add first-class support for tracking links in the WYSIWYG (TinyMCE)
editor by introducing an on/off checkbox on the link dialog.
- Improve default dummy campaign content to highlight this.
E-mails in the domain blocklist are disallowed on the admin UI, public
subscription forms, API, and in the bulk importer.
- Add blocklist setting that takes a list of multi-line domains on the
Settings -> Privacy UI.
- Refactor e-mail validation in subimporter to add blocklist checking
centrally.
- Add Cypress testr testing domain blocklist behaviour on admin
and non-admin views.
Closes#336.
- Namespace all admin UI URLs behind `/admin/*`.
This breaks the current admin UI URLs.
- Make Vue output build assets to `frontend/dist/*` instead of
`frontend/dist/frontend`.
- Namespace Vue static assets to `/admin/static/*`.
This commit reduces the cofusing and convoluted Vue+WebPack build URI
and static path schemes. In addition, it removes ambiguity in URLs
where non-UI URLs like `/public`, `/api`, `/webhooks` etc. were in the
same name space as UI URLs like `/campaigns`, `/lists` etc. Now all UI
URLs are behind `/admin/`, also simplifying security rules for proxies.
Passing `?minimal=true` to the /lists API returns all lists without
additional metadata (subscriber count) which is orders of magnitude
faster than counting subscribers per list in large DBs.
The frontend intitialization always calls the GET /lists API on load
to keep it available in multiple contexts like the new campaign page.
However, this "boot up" call does not need additional metdata. This
initialization GET /lists call now calls /lists?minimal=true.
- Add indexes.
- Refactor dashboard charts and view/click count queries.
(~10x speed bump on a setup of 7mn subscribers and 80mn views)
- Refactor get subscriber queries.
(~10x speed bump on 7mn subscribers)
- Make subscriber UI issue an equality query for email seach strings.
- Introduce a new S3 backend URL on the settings UI
- Add DB migration to populate S3 URL for existing S3 settings
- Refactor and fix URL formatting
Closes#139
- Blocklist or unsubscribe subscribers based on a bounce threshold
- Add /bounces UI for viewing bounces and in the subscriber view
- Add settings UI for managing bounce settings
- Add support for scanning POP3 bounce mailboxes
- Add a generic webhook for posting custom bounces at /webhooks/bounce
- Add SES bounce webhook support at /webhooks/services/ses
- Add Sendgrid bounce webhook support at /webhooks/services/sendgrid
Ref: https://github.com/knadh/listmonk/issues/409
- Introduce `main.appDir` and `main.fronendDir` Go compile-time flags
to hardcode custom paths for loading frontend assets
(frontend/dist/frontend in the repo after build) and app assets
(queries.sql, schema.sql, config.toml.sample) in environments where
embedding files in the binary is not feasible.
These default to CWD unless explicitly set during compilation.
- Fix the Vue favicon path oddity by copying the icon into the built
frontend dir in the `make-frontend` step.
- Refactor subimporter New*() funcs to take opt structs.
- Refactor and simplify Vue code.
- Remove redundant i18n entries and use existing ones.
- Remove redundant subimporter constants and use existing ones.
- Consider 'overwrite' option for subscription status as well.
- Write Cypress integration tests for the new feature.
- Adds `/health` as a public facing healthcheck endpoint.
- `/api/health` is meant for internal healthchecks. This endpoint in
future can serve sensitive information about Listmonk *or* can be
deprecated if there's not enough usecase.
Closes https://github.com/knadh/listmonk/issues/380
This commit disables the automatic upe check thappens immediately
on boot, giving users an opportunity disable it from the settings UI
before any remote requests are initiated. Tupdate checks happen
every 24 houfter boot.
Ref: #326
While file content (MIME) check already existed, the lack of file
extension check allowed arbitrary extensions to be uploaded and
then accessed via the static file server. For instance, a .html file
with JPG content intersperesed with Javascript.
This commit adds a file extension check on top of the MIME type check.
Campaign messages are handled by `manager` whereas test messages
were being pushed directly into a messenger skipping some campaign
related routines such as the addition of list unsub headers.
This commit exposes a new function `manager.PushCampaignMessage()`
that accepts arbitrary campaign messages that then pass through
the standard campaign message workers, thus getting the missing unsub
headers. This closes#360.
In addition, this removes the superfluous `CampaignMessage.Render()`
function which had to be mandatorily called always and makes it
implicit in `manager.NewCampaignMessage()`.
When no static assets are found on init, i.e., when a binary without
stuffbin assets are loaded, the app looks for all necessary static
files in the working dir, including the `./static/*` path which renders
the `--static-dir` flag irrelevant.
This patch gives `--static-dir`, if set, precedence over `./static/*`
when loading assets from the working dir when a binary is not stuffed
with static files.
Closes#340.
Previously, converting between formats simply copied over raw content.
This update does actual conversion between different formats. While
lossy, this seems to a good enough approximation for even reasonbly
rich HTML content. Closes#348.
- richtext, html => plain
Strips HTML and converts content to plain text.
- richtext, html => markdown
Uses turndown (JS) lib to convert HTML to Markdown.
- plain => richtext, html
Converts line breaks in plain text to HTML breaks.
- richtext => html
"Beautifies" the HTML generated by the WYSIWYG editor unlike the
earlier behaviour of dumping one long line of HTML.
- markdown => richtext, html
Makes an API call to the backend to use the Goldmark lib to convert
Markdown to HTML.
Use a dummy subscriber instead of fetching a random one from the
DB. In addition, replace the preview campaign UUID with a dummy
one to prevent clicks and views being registered against the
campaign when previewing.
The new `--i18n-dir` directory allows the loading of an external
directory of i18n JSON files, milar to have `--static-dir`
works. New languages can be added and existing language files
can be customized this way.
This commit changes file loading behaviour so that invalid or
non-existent don't halt the execution of the app completely but
merely throw a warning and continue with the default (en) lang.
Sending th optional flag as `trunue` in the POST /api/subscrirs
body will skip sending opt-iconfirmation e-mails to subscribers
and mark list subscriptions in the request a`confirmed`.
Change the behaviour where not passing attribs to the update API
overwrites the attribs with empty values. This commit changes the
behaviour so that in the absence of the attribs field in the
subscriber API, the existing value in the DB is retained.
If a user is already subscribed to an optin list but hasn't
confirmed, subscribing using the same e-mail id from the public
form now re-sends the optin e-mail while also showing an
appropriate message on the frontend rather than just saying
"subscribed successfully".
https://github.com/knadh/listmonk/issues/266https://github.com/knadh/listmonk/issues/264
The earlier approach of loading `/api/config.js` as a script on
initial page load with the necessary variables to init the UI is
ditched. Instead, it's now `/api/config` and `/api/settings` like
all other API calls. On load of the frontend, these two resources
are fetched and the frontend is initialised.
In addition to generating HTML forms for selected public lists,
the form page now shows a URL (/subscription/form) that can be
publicly shared to solicit subscriptions. The page lists all
public lists in the database. This page can be disabled on the
Settings UI.
This commit removes the Go html2text lib that would automatically
convert all HTML messages to plaintext and add them as the alt
text body to outgoing e-mails. This lib also had memory leak
issues with certain kinds of HTML templates.
A new UI field for optionally adding an alt plaintext body to
a campaign is added. On enabling, it converts the HTML message in
the campaign editor into plaintext (using the textversionjs lib).
This introduces breaking changes in the campaigns table schema,
model, and template compilation.
The default language (en) is loaded first and the selected
language is loaded on top of it so that missing translation keys
in the selected language will have the original English strings
available on the UI.
This was a ridiculous miss, where on first time installation, the
well designed default e-mail template was never installed in the
DB! I never spotted this because my local dev setup, and
surprisingly, nobody ever complained that the default campaign
template was a blank slate with no styles.
Certain SMTP hosts limit the total number of messages that can be
sent within a window, for instance, X / 24 hours. The concurrency
and message rate controls can only limit that to a max of
1 messages / second, without a global cap.
This commit introduces a simple sliding window rate limit feature
that counts the number of messages sent in a specific window, and
upon reaching that limit, waits for the window to reset before
any more messages are pushed out globally across any number of
campaigns.
Context: https://github.com/knadh/listmonk/issues/119
- Configures `max_open` and `max_idle` in default configs to `25`.
This changes the previous behaviour of connection pooling where both the
values were unset (from default config) and causes unbounded connection
limit and no connection reuse.
- Configures `db.SetConnMaxLifetime` which sets the maximum time the
connection can be reused in a pool.
- Sets `max_conn_lifetime` in default config as `5 minutes`.
Closes https://github.com/knadh/listmonk/issues/225
Lists, campaigns, and subscribers tables now support server-side
sorting from the UI. This significantly changes the internal
queries from prepared to string interpolated to support dynamic
sort params.
The link_clicks.link_id table was NULLable incorrectly. Links that
do not exist should not register a tracking entry. Fix the query
and also update the schema + migration (breaking table change).
