Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Help with TTP for fixed column table #15

Closed
jeremyschulman opened this issue Jul 27, 2020 · 9 comments
Closed

Help with TTP for fixed column table #15

jeremyschulman opened this issue Jul 27, 2020 · 9 comments
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@jeremyschulman
Copy link

Hi @dmulyalin - I am having a very challenging time trying to create a TTP template to handle this tabular input. Would you have any suggestions? My two issues is (1) handling the case where the "Name" column is empty and is instead consuming the "Status" as "Name", and (2) Handling the case for Gi0/16 - I am using ORPHRASE but TTP is only consuming the first word and then making Status "te1/1/4".

Port      Name               Status       Vlan       Duplex  Speed Type
Gi0/1     PIT-VDU213         connected    18         a-full  a-100 10/100/1000BaseTX
Gi0/2     PIT-VDU211         connected    18         a-full  a-100 10/100/1000BaseTX
Gi0/3     PIT-VDU212         notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/4                        connected    18         a-full  a-100 10/100/1000BaseTX
Gi0/5                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/6                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/7                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/8                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/9                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/10                       notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/11                       notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/12                       notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/13                       disabled     1            auto   auto 10/100/1000BaseTX
Gi0/14                       disabled     1            auto   auto 10/100/1000BaseTX
Gi0/15                       connected    trunk        full   1000 1000BaseLX SFP
Gi0/16    pitrs2201 te1/1/4  connected    trunk        full   1000 1000BaseLX SFP

I very much appreciate your help and guidance. Thank you!
@dmulyalin
Copy link
Owner

dmulyalin commented Jul 28, 2020 •
edited
Loading

This should do it for this particular piece of data

from ttp import ttp

data = """
Port      Name               Status       Vlan       Duplex  Speed Type
Gi0/1     PIT-VDU213         connected    18         a-full  a-100 10/100/1000BaseTX
Gi0/2     PIT-VDU211         connected    18         a-full  a-100 10/100/1000BaseTX
Gi0/3     PIT-VDU212         notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/4                        connected    18         a-full  a-100 10/100/1000BaseTX
Gi0/5                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/6                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/7                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/8                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/9                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/10                       notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/11                       notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/12                       notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/13                       disabled     1            auto   auto 10/100/1000BaseTX
Gi0/14                       disabled     1            auto   auto 10/100/1000BaseTX
Gi0/15                       connected    trunk        full   1000 1000BaseLX SFP
Gi0/16    pitrs2201 te1/1/4  connected    trunk        full   1000 1000BaseLX SFP
"""

template = """
<group macro="get_speed_and_type">
{{ interface | exclude("Port") }}  {{ name | re(" +") | re(ORPHRASE) | strip }}  {{ status }}  {{ vlan }}  {{ duplex | re("[A-za-z\-]+") }}  {{ speed_type | PHRASE }}
</group>

<macro>
def get_speed_and_type(data):
    speed_type = data.pop("speed_type").split(" ")
    data["speed"] = speed_type[0]
    data["type"] = " ".join(speed_type[1:])
    return data
</macro>
"""

parser = ttp(data, template)
parser.parse()

print(parser.result(format="tabulate", headers="interface,name,status,vlan,duplex,speed,type")[0])

# will print:
# interface    name               status      vlan    duplex    speed    type
# -----------  -----------------  ----------  ------  --------  -------  -----------------
# Gi0/1        PIT-VDU213         connected   18      a-full    a-100    10/100/1000BaseTX
# Gi0/2        PIT-VDU211         connected   18      a-full    a-100    10/100/1000BaseTX
# Gi0/3        PIT-VDU212         notconnect  18      auto      auto     10/100/1000BaseTX
# Gi0/4                           connected   18      a-full    a-100    10/100/1000BaseTX
# Gi0/5                           notconnect  18      auto      auto     10/100/1000BaseTX
# Gi0/6                           notconnect  18      auto      auto     10/100/1000BaseTX
# Gi0/7                           notconnect  18      auto      auto     10/100/1000BaseTX
# Gi0/8                           notconnect  18      auto      auto     10/100/1000BaseTX
# Gi0/9                           notconnect  18      auto      auto     10/100/1000BaseTX
# Gi0/10                          notconnect  18      auto      auto     10/100/1000BaseTX
# Gi0/11                          notconnect  18      auto      auto     10/100/1000BaseTX
# Gi0/12                          notconnect  18      auto      auto     10/100/1000BaseTX
# Gi0/13                          disabled    1       auto      auto     10/100/1000BaseTX
# Gi0/14                          disabled    1       auto      auto     10/100/1000BaseTX
# Gi0/15                          connected   trunk   full      1000     1000BaseLX SFP
# Gi0/16       pitrs2201 te1/1/4  connected   trunk   full      1000     1000BaseLX SFP

How it works - lots of regexes:

  • exclude("Port") - this is to exclude headers string from match results
  • {{ name | re(" +") | re(ORPHRASE) | strip }} - this is to inform TTP that match variable regex should be " +" or "ORPHRASE" pattern matching any of this: spaces, single word, phrase; strip function is to remove spaces
  • {{ duplex | re("[A-za-z-]+") }} - this is to match any alphabetic symbols and hyphen "-" character (anything but digit or space), to stop regex from consuming integer "1000" as a "status" match variable
  • {{ speed_type | PHRASE }} - this is to match any combination of speed and type columns, combining them in a single match
  • macro="get_speed_and_type" - this is to further process "speed_type" match results to extract speed and type values by splitting/joining subject match string

As you can see, this template, does not seems to be simple. Imagine how exciting it would be to troubleshoot any issues with it. For example, this slight variation will not be matched:

Gi0/16    pitrs2201 te1/1/4  connected    trunk        full   1000  1000BaseLX SFP

two spaces between 1000 and 1000BaseLX SFP

While for humans it is easy to make sense out of tabular data, its not the easiest task to consume semi-tabulated data by program if its not regular, it would be better to use some other show commands output.

For instance, this:

SW1#show int status
Port      Name               Status       Vlan       Duplex  Speed Type
Et0/0                        connected    routed     a-full   auto RJ45

and this:

SW1#show interfaces
Ethernet0/0 is up, line protocol is up (connected)
  Hardware is Ethernet, address is aabb.cc00.2000 (bia aabb.cc00.2000)
  Internet address is 192.168.1.152/24
  MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, Auto-speed, media type is RJ45
...

contains same information. However, it would be easier to extract data from latter sample because its more regular:

<group>
{{ interface }} is {{ ignore }}, line protocol is {{ ignore }} ({{ status }})
  {{ duplex }}, {{ speed }}, media type is {{ type }}
</group>

@dmulyalin
Copy link
Owner

dmulyalin commented Jul 28, 2020 •
edited
Loading

Added new match variable indicator _headers_ (commit: 3363928) to do some regex magic and dynamically form variables, as a result this template should give same results as in previous comment:

template = """
<input load="text">
Port      Name               Status       Vlan       Duplex  Speed Type
Gi0/1     PIT-VDU213         connected    18         a-full  a-100 10/100/1000BaseTX
Gi0/2     PIT-VDU211         connected    18         a-full  a-100 10/100/1000BaseTX
Gi0/3     PIT-VDU212         notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/4                        connected    18         a-full  a-100 10/100/1000BaseTX
Gi0/5                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/6                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/7                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/8                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/9                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/10                       notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/11                       notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/12                       notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/13                       disabled     1            auto   auto 10/100/1000BaseTX
Gi0/14                       disabled     1            auto   auto 10/100/1000BaseTX
Gi0/15                       connected    trunk        full   1000 1000BaseLX SFP
Gi0/16    pitrs2201 te1/1/4  connected    trunk        full   1000  1000BaseLX SFP
</input>

<group>
Port      Name               Status       Vlan       Duplex  Speed Type   {{ _headers_ }}
</group>   
"""

Key thing - {{ _headers_ }} string must match exactly the original data to properly extract results, as TTP uses exact number of characters to form column regexes, i.e. it produces this:
\\n(?P<Port>.{10})(?P<Name>.{19})(?P<Status>.{13,})(?=\\n)
based on this string:
Port Name Status {{ _headers_ }}

checkout this test for complete example:

ttp/test/pytest/test_headers_indicator.py

Line 10 in 3363928

def test_headers_indicator():

@jeremyschulman
Copy link
Author

jeremyschulman commented Jul 28, 2020 •
edited
Loading

@dmulyalin - wow the headers addition is awesome! I am going to try this out today!
Thank you very much for your help and updates!!

Looking at the test code example, I am guessing that I can use the headers indicator for any fixed-width-single-line table output? Is that correct? If so, this new feature will be quite useful is so many use-cases!!

@jeremyschulman
Copy link
Author

I am curious how I would work with a text able that has a "multi-line" header like this:

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------

@jeremyschulman
Copy link
Author

jeremyschulman commented Jul 28, 2020 •
edited
Loading

Confirmed that the _headers_ indicator works for the use-case. Now trying to figure out how to use it with the multi-line use-case ☝️

😄

@dmulyalin dmulyalin added enhancement New feature or request help wanted Extra attention is needed labels Jul 28, 2020
@dmulyalin
Copy link
Owner

dmulyalin commented Jul 28, 2020 •
edited
Loading

I am guessing that I can use the headers indicator for any fixed-width-single-line table output? Is that correct?...

yes, that is correct - expect any fixed-width-single-line table to work fine.

@dmulyalin
Copy link
Owner

I am curious how I would work with a text able that has a "multi-line" header like this:

--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------

Need to clean up results to exclude unnecessary matches:

template = """
<input load="text">
--------------------------------------------------------------------------------
Port      Name               Status       Vlan       Duplex  Speed Type
--------------------------------------------------------------------------------
Gi0/1     PIT-VDU213         connected    18         a-full  a-100 10/100/1000BaseTX
Gi0/2     PIT-VDU211         connected    18         a-full  a-100 10/100/1000BaseTX
Gi0/3     PIT-VDU212         notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/4                        connected    18         a-full  a-100 10/100/1000BaseTX
Gi0/5                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/6                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/7                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/8                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/9                        notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/10                       notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/11                       notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/12                       notconnect   18           auto   auto 10/100/1000BaseTX
Gi0/13                       disabled     1            auto   auto 10/100/1000BaseTX
Gi0/14                       disabled     1            auto   auto 10/100/1000BaseTX
Gi0/15                       connected    trunk        full   1000 1000BaseLX SFP
Gi0/16    pitrs2201 te1/1/4  connected    trunk        full   1000  1000BaseLX SFP
</input>

<group macro='clean_up'>
Port      Name               Status       Vlan       Duplex  Speed Type   {{ _headers_ }}
</group>   

<macro>
def clean_up(data):
    if "----" in data["Port"]:
        return False
    return True
</macro>
"""

added this test as example:

ttp/test/pytest/test_headers_indicator.py

Line 194 in 4fbca6b

def test_headers_indicator_3():

@jeremyschulman
Copy link
Author

@dmulyalin - this is awesome, thank you.
Will you be cutting a new release (?0.4.0) anytime soon?

@dmulyalin
Copy link
Owner

Yeah, have plans for new release, but need to update documentation and add more tests for new features.

@dmulyalin dmulyalin mentioned this issue Sep 18, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jeremyschulman @dmulyalin