Public consultation on plan for mod distribution system

Note: Please read this in entirety before posting anything. It is primarily aimed at those that intend to develop mods for KAG but is also relevant to those that would use them, both as players and server owners (albeit to a lesser extent).

Modding is going to be huge in the full version of KAG - it'll be an endless source of new content. However, there are some limitations in the current system for distributing mods. I intend to develop a sophisticated system that makes the process of getting mods (and updates to said mods) out to players as painless as possible for developers.

Overview

In order to explain how it will work I will first walk through a typical workflow from the development of a mod to the mod actually being used by a client, before exploring some aspects in more detail:

Note: I am going to skim over the majority of the technical details, as my focus at this point is on giving a high level overview of the proposed system.

1. A developer starting a new mod will register an entry for said mod with the API - they will include a unique name as part of this registration.
2. A mercurial repo (note: I would also like to support git, though it is not promised) will be auto-generated for the project (the URL for accessing the repo will follow a standard format that can be constructed from the mod details).
3. The developer will then set additional settings on the mod. What settings will be available will be discussed further down, though they are primarily concerned with access control.
4. The developer can then commit their code to the repo.
5. At any point the developer can generate a packaged version, registered with a version number (following the format [major].[minor].[patch], with [patch] as optional). In response, a tar.gz of the mod's files at that point in time will be generated, which can then be downloaded via a standardised URL constructed from the mod's identifier & the version number.
   In order for the version to be generated they must include a modinfo.json metadata file, which will include the mod’s registration info (if it does not match the details registered with the API it will be rejected), a list of dependencies (mod identifiers with specific version numbers) and the most recent version number of the game that the mod is compatible with.
   Possibly need some flags on versions for whether they are things like alpha, beta, RC, release and maybe latest/default.
6. A server owner configuring their server to use the new mod would not have to download the files themselves. Instead there will be a specific configuration file on the server in which a list of mod identifiers can be entered (optionally with version numbers too). When a server starts/restarts it will download, unpack and load the packages for the listed mods, including any dependencies. This will default to using the latest version available or alternatively will get a specific version if a version number was specified. It will not download the package if the server already has it and it has not been modified locally (more on this later).
7. When a client is trying to connect to a modded server it will request the list of active mods for the server from the API. It then asks the API for the details of each mod and checks if it needs to download any of them. If it does the client will be notified and if they confirm that they are happy with that then the client will download, unpack and load the mods before connecting to the server.

Mod settings

The settings I intend to support on mods are:

• Description
• Access control tied into our account system, for several things:- Access to viewing/downloading the source (all vs restricted list)- Commit access (restricted list)- Package download permission (all vs restricted list)
• Server ACL that uses the new API key system that I’ll be developing in advance of this mod system - essentially working in the same way as access control for users, for restricting which servers are allowed to run the mod
• Whether the mod will be ‘publicly listed’ or not (whether a GET to https://api.kag2d.com/mods will include the mod in the returned list)
• Option to disable checking for local changes to mods, to allow rapid tinkering
• Options to reject versions if the specified KAG version number or dependency version numbers are not the most recent
• Possibly support open/closed source mods (see further down for details)
• If closed source mods go ahead then premium mods may be an option too

Other notes

Some of the following are ideas that haven't been fleshed out yet or not guaranteed to be included, some are just 'extra info':

• The system will be entirely exposed by the API. This means, it will be possible for people to build their own web interfaces to it, and indeed I would love to find someone to build a website that allows for managing and searching mod (similar to how garanis built his excellent server browser site) using the existing SSO infrastructure for authentication & a new API key system I will be developing. The basic idea behind the API key system is that you will have to sign in via SSO and then give the site specific permissions (e.g. ‘manage my mods’) and from then on the site will be allowed to make specific kinds of API requests on your behalf. Note: If you are a web developer that would be interested in working on a web front end to this service then I'd love to hear from you to discuss it - PM me.
• Ideally there will be a +1/-1 reddit-style mod rating system that will (hopefully) allow for community moderation of mods. The specifics of this have yet to be worked out.
• Servers will be allowed to disable the API method of downloading mods, at the cost of not being available in the main server list by default (you will have to enable a filter to show ‘unmanaged mod’ servers). They will then use the current method of direct download from server to client or instead require that clients download mods externally and independently.
• I'd quite like to allow ‘closed source’ mods. Currently we distribute mods as their raw scripts, which are then compiled into AngelScript bytecode at runtime. If there was sufficient demand I would be interested in allowing the option to have a closed source mod in which AngelScript is run in order to compile the scripts to bytecode and save them as files whenever a new mod version is created. These bytecode files would then be what is downloaded and loaded by the servers/clients.
• I'd like to collect mod usage stats so we can track how many servers + players used each mod over time
• Something I'm particularly excited about with this system is the possibility to create 'framework' mods, thanks to the dependency + version system. Someone could, for example, create a mod that provides the basic building blocks of an RPG (XP, quests, NPCs, etc.). Other mods could then build on that mod by listing it as a dependency (and likely fixing the dependency to a particular version in order to ensure that the new mod won't break if the framework mod is updated). You'd then potentially have many RPG mods that were all written in a fraction of the time because they had this framework to build on. An alternative to framework mods is that you could create 'mod packs' that combine many smaller mods into one (again using dependencies) in order to provide a more complete experience.

I would love to hear your feedback on this system and will appreciate any criticism provided that it is constructive. I'm especially keen on what potential mod developers think of the proposal, though I'd also like to hear what regular users think about it.

 

1. +1/-1 refers to a reddit-style system of upvoting, rather than something that gets the average score calculated. The specific algorithm that will be used to determine score has not been decided on yet but it will be based on the total up/downvotes.

2. I'd like extensive sorting like that, though whether or not that is something that the API should handle itself or something that is implemented in a third party website I haven't decided on yet.

3. Yes, all mods should register with the API, though you might want to question why you need such insignificant mods. If you just want a small tweak to an existing mod then a better solution might be to offer a patch to said existing mod. You'll still be able to have other smaller mods if you choose not to use the API-centric system though, but you'll be listed as a server running 'unmanaged mods'.

EDIT: added a new point on the end of the other notes section that might be of interest, Toast.

 

All mod settings would still apply if the mod was a dependency - if it was closed, then if it was a dependency it would be downloaded as bytecode, if it was restricted to specific servers then mods that used it as a dependency would also be restricted to said servers.

I am thinking about premium mods, but I don't want to explore that yet - its far too early to get into details on that. However, if it were to happen is that all players would have a record of mods that they have purchased and servers running premium mods would only allow people that have purchased said mods to join them.

You mention a different distribution method... What did you have in mind and why do you think this system would not satisfy your needs (if it supported closed source, server-restricted, premium mods)?

 

Incidentally, if anyone is (or knows) a web developer whom would be interested in working on a site that functions as a front end to this service (either as just the mod management system, a mod search system or both) then PM me as I'd love to discuss it with them. We won't be looking to pay anyone to do this but I'm comfortable with them making money through the site provided that the basic service (i.e. allowing mod management or mod searching) is free. Provided that it is of sufficient quality, such a site would be officially endorsed.

 

Without downvoting it won't serve its intended purpose of providing community moderation though :/

 

It doesn't have to reset for what is worth. User would be allowed only to upvote or downvote at time. So if one changes mind and changes his vote then it will work fine. At least I think it should be done this way.

Overall, Tom, it sounds amazing (like it did when you first introduced the idea to us). Are you sure that Dave won't have time to work on the website? It still far future. (:

 

Thanks, this is exactly what I made this thread for :D

1. Searching in the API call will be possible in the same way that it is currently possible when searching for servers
2. Yeah I'd prefer git too. However all our own (KAG Team) development tools are built around mercurial, hence why it is getting first dibs. Full disclosure: I would really, really like git in there so I'm pretty sure I'll get around to it eventually, its just that supporting a second VC system isn't top of the priority list. When I do get around to it I'll probably implement some sort of automated transition system that'll do its best to preserve commit history if you choose to swap repo.
3. It will follow all of the same conventions as the current API, so yes it'll be using JSON. There may be some sort-of-exceptions in that where conventions need improvement the same improvement will be applied to elsewhere in the API. However, we need to implement proper version support in the API before that happens - which is itself planned to be implemented before development of this mod distribution system gets underway.
4. I like this and will likely include it as one of the improvements I retroactively apply to the server side of the API :)
5. The general idea is that mods will be verified against the mod stored with the API by simple hash comparisons. The unaltered game code won't allow clients/servers to run scripts that haven't been confirmed with the API as being the same... However there is still the issue of how to stop hackers from subverting this process by altering the game code. We have plans for this but I am not going to discuss it in a public forum :)
6. Download and usage stats are the main 2 things I want, we'll see what else seems appropriate closer to the time.
7. Compatibility with KAG itself is a good point that I hadn't really given much thought. If I make that something that is managed in the API then it will probably just be an extra bit of meta data against each mod version. For it to be something that is altered by the community it would either have to be something similar to how ratings are implemented (though I'm really not sure how this would work) or simply a service provided by an external website.
8. Agreed.
9. Yep, these echo similar thoughts I'd had. I'm particularly interested in doing things like 'featured' mods once the mod ecosystem has matured a bit.
10. The AngelScript bindings don't exist to perform anything destructive like that - though I've had one or two ideas of how they could be abused to a lesser extent. As such we will have moderators, but a major reason for having a rating system is that things that get downvoted into oblivion will be examined more closely to see what people are complaining about. Additionally I'm considering some sort of separate 'report abuse' system, for cases when mods are specifically malicious.

EDIT: Had some other thoughts, particularly re: point 7 (and our IRC discussion), and the plan is getting some revisions. Thanks :D

 

I don't care who works on it so long that someone does (and they do a good job).
If Dave wants to then I'm perfectly fine with that :D

 

What Furai said - your vote can be changed at any time. New votes will not be cumulative - you have one vote per mod, which is replaced if you change it from down to up or vice versa.

 

I've considered things like that. That's why we will still continue to support mods that are not API-managed (servers running them will just have an 'unmanaged' flag on them in the server browser that will be filtered out by default).
But the API is going to enforce the use of repos (and because it can't feasibly support all repos its just going to enforce the use of the ones we like :p). If you don't want to use the repos like repos, that's up to you (you could develop mods independently, testing them as unmanaged mods and then only upload changes to the repo in a single, massive commit if you so wished - though I certainly don't advise it).

And I'm definitely not doing anything that requires admins to manually add content to an official repo.

 

I know that Soldat's modding system was used to download viruses to user's computers. This API system will only allow people to download approved mods, but won't KAG devs be automatically responsible for any mod on the API this way? IE, won't people have the ability to start a law-suit against the responsible KAG devs if one of the mods contains an exploit?