Overview¶
One task called youNotify contains the following plugins:
youtube, an input plugin configured to monitor a user’s subscriptions. 10 videos are fetched from each subscribed channel every 2 hours.filter, a mutator configured to discard any video that contains the word “live” in the title.deduplicate, a mutator that has no configuration.pushbullet, an output plugin configured to send a notification to the user’s smartphone every time a new video is found.emailDigest, an output plugin configured to gather all information and release it every day at 18:30 in a single email, sent to the user’s email address.
The data fetched from Youtube is packaged as a series of distinct facts. Each fact contains metadata about a single video.
Facts are Egeria’s atoms; they can be manipulated and changed, or thrown away, but one atom will always remain one atom. It cannot be divided.
Facts are Knowledge. Knowledge is learned by input plugins.
Knowledge learned by youtube is then processed by the filter mutator. Some of that Knowledge will be thrown away.
Whatever Knowledge survives the selection will be handed over to deduplicate, which will automatically remove any fact that has been processed successfully in the past.
Since after deduplicate there aren’t any other mutators, any fact left will be fed to all of the output plugins in the task.
pushbullet will process each fact one at a time, acting on it the only way it knows: it will use some of the metadata to tell Pushbullet to send a notification to the user’s smartphone. pushbullet will not retain any facts; once used, they are lost. If something goes wrong, they are lost anyway.
emailDigest will instead accumulate Knowledge. Accumulated Knowledge is called Wisdom; a unit of Wisdom - which usually contains several facts but can be empty - is called an experience.
According to its configuration, every evening at 18:30 emailDigest will use its experience to generate an email that will be sent immediately. It will then forget its current experience and start accumulating a new one.
Since the youtube plugin always learns about the latest 10 videos from each subscribed channel and most channels will upload only 1 or 2 videos in the entire day, the task will be dealing with a lot of duplicates. The deduplicate mutator makes sure that only new facts will be handed over to the output plugins, but there are limits. Currently, Egeria will remember a fact for 6 months; once a fact is that old, it will simply disappear. Should youtube learn about that fact again 7 months later, deduplicate will let it through.
If one or more plugins fail, the fact being processed at that time will instead be forgotten entirely, as will any facts that are filtered out on purpose by any plugin (usually mutators). Those facts will find their way into the task again and again, until the input plugins stop learning them.
A task can have any number of plugins of any kind. It can even miss input (or output) plugins entirely but... it won’t do much.
Example: a task¶
Let’s go for something a little simpler. This task learns about Youtube videos; it will only retain facts that are at most 2 days old, it will reject duplicates and send notifications via Pushbullet.
{
system: {},
tasks: [{
name: 'youNotify',
chain: [{
plugin: 'youtube',
plan: '0 */2 * * *', //every 2 hours
configuration: {
mode: 'subscriptions',
limit: 10,
identity: 'gglMyName'
}
},{
plugin: 'filter',
configuration: {
'youtube:video:publishedAt': {after: [-2, 'days']}
}
},{
plugin: 'deduplicate'
},{
plugin: 'pushbullet',
configuration: {
identity: 'pbMyName',
tag: "some pushbullet tag",
mode: "link",
title: "{{youtube:channel:title}} - {{youtube:video:publishedAt!!MM/DD/YYYY}} - {{youtube:video:title}}",
body: "{{youtube:video:description}}"
}
}
}]
}
While all plugins are placed in a chain, only the order of mutators matters; input plugins such as youtube will just learn facts whenever they run and output plugins will always receive facts after they have gone through all of the mutators.
On the other hand, the order of mutators does matter; in this example placing the filter plugin, which forgets facts based on their own properties, before the deduplicate plugin, which hits the database once for every fact, is more sensible than the opposite (which would run more database queries, all of them useless).