[Request] Menu Option Selection for Scenarios

[Much ThiccParticipant]

I would like to request a scenario feature for Miqo: optionSelect()

example: *Within systems menu* optionSelect(‘Exit Game’)

There are many situations where being able to select something specific from a menu could be really useful. Leves, quests, menuing thru vendors, etc. I know there is already a framework that exists within Miqo currently such as deliverGCgear() and deliver(Collectables) that menus through the dialogue options at a faster, more effective rate than anything we as users could manually write in a scenario. I also know that the teleport() as well as the craft(‘Specific Recipe’) function works by writing our own specific input and miqo finding it.

I think it would serve as a great long term addition to the code to be able to search through a menu that is already open for a specific dialogue option and select it. Given that there are already several similar functions in place, I also am convinced this feature wouldn’t be difficult to implement whatsoever.

Edit: Further, I think it would serve Miqo users better to have this catch all function to select from a menu thats already open, rather than expect the developers to make a code sequence for a bunch of different specific dialogue options in different menus, like leves for example.

• This topic was modified 3 years, 7 months ago by  Much Thicc.

[MiqobotKeymaster]

Yes, this feature is already planned as a part of Scenario Engine updates: Upcoming Features

However, there is no framework that is capable of interacting with arbitrary game windows. Each of the functions you mentioned (deliverGCGear(), deliverCollectables(), teleport(), recipe()) incorporates a separate algorithm which is compatible only with one specific game window. These algorithms will not work for anything else.
In addition to that, the function deliverCollectables() includes 3 different algorithms in order to support 3 different UI layouts:
– Rowena’s House of Splendors
– Ishgard Restoration
– Diadem Resource Inspection
If it looks simple from the outside, it doesn’t mean there is a simple piece of code behind it.

Each game window is completely different, they have distinct memory structures and UI layouts, and each of them requires a custom sequence of keybinds. A simple generic algorithm for this problem does not exist.
Even if we implement this feature in the form of a single function, we will have to research all possible game windows separately and implement different algorithms for each of them. The amount of work will be the same as implementing multiple functions.

So yes, your feature request is accepted.
But please don’t expect it to be a generic function that will support everything, because it’s not possible.

[Much ThiccParticipant]

I only have a loose understanding of programming but I’m curious what’s stopping you from creating an internal index of dialogue options once the command is reached in the scenario, then miqo searching internally for the text populated from the data collected based on the menu that’s currently open.

I guess I see what you mean considering how many different dialogue boxes there are, but given the limited number of actual commands it takes to get to any one selection, and in general there are broad menu options that could be included in tandem commands like an accept() function, I feel like I’m missing a key part of how the program works in terms of the actual data miqo is looking at at any given moment

[MiqobotKeymaster]

Much Thicc wrote:

I only have a loose understanding of programming but I’m curious what’s stopping you from creating an internal index of dialogue options

The non-linearity of internal data structures.

What you see on the game screen is only a visual representation of the actual data. It is generated by the graphics algorithm via multiple processing stages. Even if it looks like a simple list of options on the screen, it may in fact be an associative container based on red-black tree implementation. It may not even have a container but be a procedurally generated data stream that doesn’t have any structures behind it.

Miqobot is not capable of reading the text directly from the screen, if this is what your question is about.
This would require Optical Character Recognition algorithms, but this technology is extremely slow and would not be able to perform in a dynamic environment such as the video game.
Therefore accessing and analyzing internal data structures directly in the game memory is the only way to implement gameplay automation.

[MiqobotKeymaster]

If you are still not convinced, here is a simple example.
Imagine we collect a few samples of Rarefied Gyr Abanian Mineral Water and deliver them to a collectable appraiser.
When we open the Collectables Exchange window, the game shows us this:

The item we wish to deliver is level 61. But it doesn’t even appear on the screen to begin with.
In order to access it, we have to scroll the list down, click on “Lv. 61-70” group, then scroll down again, and only then we will see it.

But if you activate deliverCollectables() from a scenario, you may notice that Miqobot does not check every possible group every time the window is opened. She knows exactly what items we have and what groups must be clicked in order to access them.

• When there are no items to deliver, Miqobot simply closes the window.
• When there are items not visible on the screen, she goes directly to required groups and selects the necessary lines.

Without reading the internal data structures, such behavior would be impossible. Miqobot would have to painfully scroll through every line and analyze every single word and number on the screen in order to identify if there is anything to deliver at all. So instead of staring at empty window for a few seconds, she would be stuck for 3-5 minutes verifying each possible option without doing anything useful.

And please believe us that the data structure for Collectables Exchange window is so complex that it took our developers several days to reverse engineer it and then several days more to implement UI navigation algorithms.
Considering that deliverCollectables() contains 3 different features each of which has underwent over 10 iterations of improvements and bugfixes, you are in fact tapping into 2 months of development every time you use this function.
All for a short sequence of steps to click the correct buttons.

[Much ThiccParticipant]

I really appreciate the time you took to respond and clarify as well as answer some questions I’ve had.

Funny enough, even after you’ve described this, I still don’t see how Miqo isn’t capable identifying the names of dialogue options as its presented to the player, because what I would assume is that at some point the actual text in any given menu that is displayed to the player would be detectable by Miqo in some form right? Also to be clear, I’m not referring to Miqo actually ‘reading’ the screen. I’m suggesting that at some point the text, such as ‘Limsa Lominsa’ in the teleport menu, exists somewhere in the data that Miqo takes from the game.

I understand collectibles is a difficult menuing UI to cover, but a better example would be teleportation. How is Miqo able to identify ‘Limsa Lominsa’, or even better, something like ‘The Crystarium’ which is pages and pages down the menu, if Miqo isn’t simply ‘searching’ for the term within the teleportation menu? Correct me if I’m wrong, and I’m sure that I am, but how it seems to operate is similar to using a search function here on the forums or anywhere else, really. My input is ‘The Crystarium’, and Miqo then uses the menuing keybinds to scroll down the teleport list until it reaches a matched query.

If the above actually functioned the way I described, that’s pretty much what I’m suggesting, just for any menu.

So for example, lets say I want to go to my apartment. In the scenario, I would write the keybinds to actually go to the front door of the apartment building and open the menu, but then use optionSelect(‘Go to your apartment.’), then Miqo, similar to the way I assume menuing through teleportation works, would go to that dialogue option.

I know that might sound redundant, but there’s tons of elaborate scenarios where bad server ping happening even once throws off the entire menuing process. It would make menuing more precise, on top of potentially being faster overall (as opposed to manually typing every keybind for menuing, along with pauses)

[MiqobotKeymaster]

Much Thicc wrote:

I would assume is that at some point the actual text in any given menu that is displayed to the player would be detectable by Miqo in some form right?

It’s not detectable. And this is the main problem.
If it was, we would have already implemented this function a long time ago.

Much Thicc wrote:

I’m suggesting that at some point the text, such as ‘Limsa Lominsa’ in the teleport menu, exists somewhere in the data that Miqo takes from the game.

The text data stored in the Teleport window looks like this:

hingashikuganekugane100othardthe ruby seatamamizu333the ruby seaonokoro357yanxianamai445yanxia...

You are correct, it is detectable in some form. But this form is useless for any kind of decision making.
Besides the fact that this data doesn’t have any line breaks, it also has no indication of whether a certain piece of text is selectable. For example, “Hingashi” and “Othard” are group names which should be removed from the list of selectable options beforehand. And for the rest of text, there is no information of where that data is located on the screen. In order to select “The Ruby Sea”, do we have to move cursor at line #2, #6, or #9?

Much Thicc wrote:

How is Miqo able to identify ‘Limsa Lominsa’, or even better, something like ‘The Crystarium’ which is pages and pages down the menu, if Miqo isn’t simply ‘searching’ for the term within the teleportation menu?

By reading the internal data structures.
Here is the structure that Miqobot reads when accessing the Teleport window.

6F 00 00 00 64 00 00 00 74 02 00 00 00 00 00 00 01 65 58 02 69 00 00 00 4D 01 00 00 65 02 00 00

This is an example of a data stream mentioned earlier. It’s not even located in the Teleport window, this data is stored in a dynamic memory container which is populated by another part of the game client.
By grouping and decoding the stream chunks correctly, Miqobot is capable of identifying a numerical aetheryte ID and a numerical zone ID, which are then converted to the correct line index to select. And when you type “teleport(Limsa Lominsa)” in Scenario Engine, Miqobot uses a hardcoded database of all possible teleport locations which link the text data to numerical IDs.

On top of that, please remember that Miqobot must be functional in all 4 game languages.
So when you type “teleport(リムサ・ロミンサ：下甲板層)”, she uses the same database to convert a Japanese name into the same numerical ID and select “Limsa Lominsa” in any language. This conversion does not work for all features, of course, but we are doing our best to make it work where possible.

[Much ThiccParticipant]

I see, that totally answers my question. It makes a lot more sense now. Thank you so much for the clarification and insight. I appreciate the work you all do and I’m very happy with my experience using Miqo overall.

[Author]

Posts